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Software Development Partnership Program 



I. Background 

Program Description. Between 1986 and 1988, the California Department of Education (CDE) 
awarded grants from Assembly Bill 803 funds to seven software publishers for business 
partnerships to develop technology-based programs to help meet instructional needs of California 
students. The matching grants were intended to provide “seed money” to encourage publishers 
to contribute investment capital in return for guidance from the CDE. The technology products 
were intended to address areas in the California Curriculum Frameworks that were lacking in 
high-quality software and to fill the “holes” that had been identified by the Technology in the 
Curriculum (TIC) projects. The products were also intended to be models for 
publishers/producers to stimulate the development of technology-based instructional materials 
that were aligned with the California Curriculum Frameworks. In addition, the CDE believes 
that the availability of quality technology-based instructional materials (i.e. aligned with the 
California Curriculum Frameworks) will encourage schools to purchase appropriate hardware. 
The products to be developed by this program of business partnerships were to be made available 
at a discount to California schools and the state would receive royalties on out-of-state sales. 

Six projects were established in 1986 between the CDE and software publishers to produce 
instructional courseware in mathematics, science, and history-social science. Field tests of the 
programs were conducted in school districts by the software publishers in concert with CDE 
staff. Three of the original projects were subsequently cancelled and three were completed. The 
State is receiving royalties on the out-of state sales of the products. 

A seventh project, Science 2000, was initiated in 1989, also with funds from AB 803, and was 
completed in 1991. The titles of the seven projects, the primary contractors (and collaborative 
partners), grade levels, and subject areas addressed were as follows: 



Project Title 


Publisher 


Grade 


Subject Area 


GTV: A Geographic 
Perspective on 
American History 


National Geographic 
Society (Apple 
Computer, Inc., and 
Lucasfilm Ltd.) 


7-12 


History-Social Science 


Adventures in 
Mathland 


Mindscape 

(Picodyne Corporation) 


K-6 


Mathematics 


MECC World 
GeoGraph 


MECC 


6-12 


History-Social Science 


Explorations in 
Middle School 
Science 


Jostens Learning 
Corporation 


6-9 


Science 


Exploring Matter 


C&C Software 


9-12 


Science 


Science and Society 
Series 


Advanced Ideas 


9-12 


Science 


Science 2000 


Decision Development 
Corporation (National 
Geographic, Dinamation 
Apple, IBM, and 
Pacific Bell) 


7 


Science 
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Legislative Authority. AB 803 authorized the Superintendent of Public Instruction to “contract 
for the research and development of educational software in accordance with priorities 
established to meet pupil needs.” 

Authorization for software development project support has been continued under the current 
legislation, AB 1470, the Farr-Morgan-Quackenbush Educational Technology Act of 1989. 
Section 51873.5 states that the duties of the Educational Technology Committee shall include 
“Advising the State Board of Education and the Superintendent of Public Instruction, pursuant to 
this article, on the allocation of grants for educational software research and development.” 

II. Planning and Restructuring 

Program Planning. Each of the first six software development projects funded by the CDE was 
required to prepare plans for (1) designing a software product, (2) implementing the project, and 
(3) marketing and distributing the product. The request for proposals (RFP) for these projects, 
issued in 1986, provided broad guidelines for addressing curriculum content objectives, selecting 
technology applications, product testing, and technical specifications (CDE, 1986). The projects 
were given considerable latitude in proposing products to fill the “holes” in the curriculum. 

In contrast, the RFP for Science 2000, set specific parameters for addressing topics from the 
middle school science curriculum, utilizing a variety of educational technologies, and product 
field testing with students representative of the diverse populations in the state. 

Project Planning. The software development plans of the seven projects, particularly that of the 
contractor for Science 2000, drew upon the rationale for the program expressed in the RFPs for 
identifying the needs to be addressed by the projects, to formulate goals and objectives, for 
criteria to identify the members of curriculum and technology advisory groups, and to link 
marketing plans with other state educational improvement initiatives. 

III. Program Description 

Program Purpose. The rationale for the software development program expressed in the 1986 
RFP observed that the holes in the curriculum existed “because software and video materials are 
expensive to develop and risks are high.” It was noted further that: 



“Sales and profits are difficult to predict; consequently content areas with the 
highest probability of return on investment are selected for development. Large, 
broad-scoped programs using advanced technology, such as interactive laserdisc, 
are risky due to large development costs and a small installed base of hardware. 

The State Department of Education intends to share in the risk of developing these 
software and video products by forming “partnerships" with publishers. In such 
a partnership, the State shares the development costs, in return for preferential 
pricing of products for California schools and/or royalties on sales. The State 
Department of Education will not develop the software, but will help to fund its 
development by publishers. The State will not own the software, but will negotiate 
rights to obtain copies for California schools and receive royalties to recoup its 
share of development costs. This investment of the State’s is viewed as an 
“ advance ” against future royalties from the products, and a necessary strategy to 
bring to the marketplace products which are state-of-the-art, meet California’s 
expectations for curriculum content, and which would not otherwise be readily 
selected for development fey publishers." 
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Rationale. The rationale in the second RFP, issued in 1988, was somewhat more direct in 
stating that the grant was “intended to provide ‘seed money’ to encourage the initial coordination 
and organization of materials and programs that better meet the needs of today’s students and 
teachers. It is expected that the organizations joining in this effort will contribute a major portion 
of the investment capital, and, in return will receive guidance from the State Department of 
Education.” 

Project Objectives. Each of the seven software development projects proposed courseware that 
addressed curriculum topics and instructional objectives for specific grade levels or grade spans. 
The scope of planning for each of the projects is summarized in the following CDE descriptions: 

National Geographic Society: Technology, Geography, and the State of California, as 
originally proposed, was to be “an innovative software product which combines computer 
and optical technologies to provide dynamic tools for the teaching of a basic course on 
U.S. geography for students in grades 7-12. The partnership will draw heavily on the 
proprietary photographic resources of the National Geographic Society as well as the 
technical expertise of Apple Computer and Lucasfilm Ltd.” 

“The National Geographic Society, in collaboration with Lucasfilm, will produce Level 
One laserdiscs in the CAV mode (which provides full random access to individual images 
as well as forward, backward, and multiple-speed play). The laserdiscs can be viewed as 
a linear, well-organized collection of segments. While these segments will be laid down 
for access by Apple IIGS, more complex machines, such as the Macintosh 0, as well as 
less agile computers, such as the Apple He, also could access the laserdiscs.” 

“The completed multimedia product will consist of images (still and motion), sound, and 
text stored on optical and computer discs. Production will begin in January 1988 with a 
projected completion in 1990.” The state grant amounted to $500,000. 

Mindscape: The product, Adventures in Mathland, was to be “a comprehensive blend of 
microcomputer software, guided print activities for using commonly available 
manipulative materials, and an optional interactive laserdisc designed to meet the needs 
of math education in grades K-6. It is designed to be a series of software products, all 
sharing a common theme: a child’s adventures in a museum of mathematical discovery. 
The software will be produced both for the Apple II series and for MS-DOS machines. 
The entire product will be available October 1988.” The original CDE grant to 
Mindscape was for $82,500. 

MECC : As originally conceived by the Minnesota Educational Computing Consortium, 
MECC World GeoGraph was to be “designed as a social studies learning tool to foster 
knowledge acquisition in areas such as human geography, economics, political science, 
and demographics. This will be accomplished not through the mere presentation of facts 
and the rote memorization of figures but through direct student interaction with data.” 

“The main strength of World GeoGraph is its role in the development of student study 
skills. Study skills in acquiring, organizing, and interpreting information will underlie its 
use. Intellectual skills in comparing attributes, classifying information, formulating 
questions, drawing conclusions, and testing hypotheses will be integral to the activities 
designed for it.” 

“Students will use this tool to work with information pertaining to the United States as 
well as many other countries of the world. Students will be able to use maps to acquire 
facts and to test hypotheses and draw conclusions. The results of their inquiry will then 
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be graphically displayed, reinforcing not only which regions and countries fit certain 
criteria, but also where they are located relative to one another.” 

“The software will consist of maps and databases, and will be produced for the Apple 
IIGS machine. The product will be available April 1988.” The CDE grant was for 
$ 100 , 000 . 

Education Systems Corporation ( acquired by Jostens Learning Corporation): The 
proposal for Middle School Science described “a series of 36 computer-simulated 
laboratory activities for grades 6-9 keyed directly to what students are learning in the 
classroom.” The activities were supposed to take students about 40 minutes to complete 
and complement “live” laboratory experience. Activities provide “simulated” 
laboratories in physical science where students could act like scientists applying science 
concepts. Each lesson was to have a similar structure: 

Introduction-The introduction stimulates student curiosity and interest in the subject of 
the lesson via a brief “attention grabber.” It also provides an overview of the lesson and 
assesses the student’s prerequisite knowledge and skills. 

Laboratory-A simulated laboratory is at the heart of each lesson. It may involve 
observing, measuring, collecting and analyzing data, hypothesizing, experimenting, 
predicting, or communicating results. 

Quiz— A brief review and quiz follows each laboratory. Quizzes usually ask students to 
do something similar to what they have done in the laboratory. 

The lessons were to run on a network-managed system of MS DOS or Apple IIGS 
microcomputers with 16 color, sound, mouse interface, and excellent graphic capabilities. 
It was estimated that the CDE contract award of $167,500 would cover approximately 30 
percent of the development costs. The product was to be completed by March of 1989. 

C&C Software: Exploring Matter, a science program for grades 9-12, was to be made up 
of a series of “activities designed in an interactive environment in which students explore 
the physical and chemical properties of matter.” The activities were to be designed to 
guide student “study and exploration as they record observations, make measurements, 
and carry out investigations in either the actual physical science laboratory or in the 
simulated laboratory that is provided.” 

Designed for the Apple IIGS series of computers, the activities were to “give students an 
opportunity to study and explore science concepts and to learn basic laboratory 
techniques. The activities are selected and adapted to fit the capabilities of individual 
students. The programs automatically adjust to each student’s skill level during 
operation, record scores and information about that students current level, and use and 
add to that information on subsequent runs.” The product was to be available by 
December 31, 1988. The CDE grant for Exploring Matter was $80,000. 

Advanced Ideas: The Science and Society Series, another science program for grades 9- 
12, was to consist of four educational computer programs designed for use in home or 
schools. A four disk series of programs was to “teach the principles of the scientific 
process and the social implications of those principles in an interactive adventure format. 
The theme of the series will deal with conservation issues and will feature colorful 
animated graphics, simple but powerful applications, and fast-paced writing.” 
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“The products will incorporate databases, ecological models, and decision-making in a 
series of disks centered around the interactive conservation stories. The stories combine 
the scientific process with social decision making. The software program will make it 
possible for a teacher to efficiently deliver specific science curriculum materials.” 

Each disk in the two-disk software series was to present a story about an endangered 
species or resource. The person using the software was to be a participant in the story 
and play a pivotal role in deciding the outcome of various situations. The product was to 
be produced in MS-DOS and Apple versions by June of 1988. 

Decision Development Corporation: Science 2000 was proposed to be a “complete 
technology-based learning resources management system for teaching science in grade 
7.” Decision Development Corporation established partnerships with Optical Data, 
Interactive Media, Inc., National Geographic Society, Dinamation, and Pacific Bell to 
develop the product “aligned with the new California Science [Curriculum] Framework 
and will demonstrate how a variety of technology-based materials (e.g., computer 
software, films, video and audio tapes, disks) can be configured to provide students the 
opportunity to experience an optimum science curriculum. The product will contain 
lesson plans which integrate the teaching of the earth, life, and physical sciences. The 
hypermedia tools will provide the teacher with access to a variety of resources for 
teaching the lesson. In addition to referenced resources, the program will include original 
materials on a laserdisc and in printed teacher guides.” 

Total monies authorized were $884,380 from fiscal year 1988-89 funds to Decision 
Development Corporation to create Science 2000. The product was to be completed in 
June of 1991. 



IV. Program Implementation 



Project Completion and Cancellation. As mentioned previously, four of the seven software 
development projects were completed and are now being marketed in schools in California and 
other parts of the country. For various reasons, summarized below, three of the original projects 
were not completed; these included: 



Publisher Product Title 



Grant Amount 

Award Paid 



Mindscape Adventures in Mathland $ 82,500 $ 14,850 

Reason Cancelled: The contract for the development of the product was cancelled by 
Mindscape in June of 1989, after two years of production when the principal subcontractor, 
Picodyne Corporation, suffered a financial failure and went out of business after providing 
only the first deliverable under the contract. Mindscape officials noted that they had 
already advanced Picodyne more money than they had received from the State of 
California. 

CDE Position: The notice of contract cancellation from the officials at Mindscape was 
received with regret by the CDE. The Office of Educational Technology’s 
acknowledgement of the notice of cancellation stated that “conceptually, Mathland had the 
promise of being an outstanding product” and expressed “hope that someday we may work 
together on another project.” 
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_ . , Grant Amount 

Publisher Product Title Award Paid 

Advanced Ideas Science and Society Series $90,000 $41,480 

Reason Cancelled: Advanced Ideas began work on the Science and Society Series during 
August of 1987 while final contract negotiations were being completed. On January 28, 
1988, after extensive exchanges through correspondence and in meetings among staff of 
both the CDE and the contractor, about the quality of two sets of prototype materials that 
had been submitted for CDE review, officials of Advanced Ideas submitted a notice of 
cancellation. In a letter dated March 29, 1988, the Office of Educational Technology 
acknowledged receipt of Advanced Ideas’ cancellation and issued the CDE’s own notice of 
cancellation. 

CDE Position: The CDE maintained that Advanced Ideas had not complied with the 
contract in completing the software product. Reviewers of Science and Society Series 
prototypes had considered the materials deficient in regard to grade level, program 
structure, curriculum focus, and production quality. In a letter dated February 3rd, in 
response to the notice of cancellation from Advanced Ideas, the Office of Educational 
Technology maintained that CDE staff comments had been timely and provided 
constructive criticism of the prototype product. In its cancellation notice, the CDE also 
demanded reimbursement of $20,740 of the funds that had already been advanced for the 
second production stage of the contract because the product was not satisfactory. 

Contractor Position: In their contract cancellation notice, the officials of Advanced Ideas 
protested that the concerns expressed by the CDE about the content of the storyline of the 
prototype, the appropriateness of the product for the targeted grade level, the quality of the 
graphics, and the sequence of responses had not been provided in a timely manner. They 
also maintained that CDE staff had not followed through on commitments and that 
comments provided in earlier meetings had not been helpful in the revision of prototype 
materials. 



Publisher Product Title 



Grant Amount 

Award Paid 



C&C Software Exploring Matter $80,000 $62,000 

Reason Cancelled: The Office of Educational Technology issued a notice of cancellation 
by the CDE of the contract on April 4, 1989, citing unsatisfactory performance as the 
reason for the termination. The contract to C&C Software had been awarded on September 
22, 1987, and the contractor worked for nearly 18 months while protracted correspondence 
was being exchanged over the quality of the experimental versions of the Exploring Matter 
materials and over payment for deliverables already made. 

CDE Position: The CDE stated that the products submitted by C&C were not satisfactory 
and showed little promise of living up to expectations or of meeting the requirements of the 
California Science Framework. In January of 1989, the Office of Educational Technology 
had advised C&C that Exploring Matter materials submitted in December of 1988 were not 
satisfactory and that the CDE would not approve of any further contract payments until the 
contractor had made dramatic improvements in the product. Unfavorable recommendations 
from the manager of the CDE science curriculum unit and an independent evaluator had 
previously been cited as reasons behind the judgment that the product would probably 
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never live up to CDE expectations. In March of 1989, the CDE advised C&C that the 
contract was to be cancelled because: (1) “the software is not what we would deem 
software of quality,” (2) “the software lacks pedagogy of quality and is not technically 
sophisticated,” (3) “the software presentation is a trivialization of the concepts being 
presented,” and (4) “the software is not a product we want to distribute to California 
schools as representative of the California Science Framework.” 

Contractor Position: Officials at C&C Software insisted that until January 28, 1989, they 
had believed that the project had been proceeding satisfactorily in accordance with a 
revised schedule agreed upon with the CDE in the summer of 1988. The company disputed 
CDE statements about corrections to flaws in the software and the provision of suggestions 
to improve the quality of the science content in the materials. The president of C&C 
argued that “communication from your end had been so poor that it could not possibly have 
made a positive contribution, and that we saw the problems that then existed with the 
project as largely the result of your attempts to guide.” 



The canceled projects were the three smallest, accounting for only about six percent of the total 
CDE expenditures. 



V. Resources to Support the Projects 

Support Factors. In addition to the AB 803 grants from the CDE, two of the software projects 
that completed their contracts were able to draw upon the resources of “high-powered” partners 
in the software development industry. The National Geographic Society was both the prime 
contractor for one project (GTV: A Geographic Perspective on American History) and a partner 
in another (Science 2000). GTV also benefited from the resources of Apple Computer and 
Lucasfilm Ltd. The Decision Development Corporation was able to draw upon the considerable 
creative and financial resources of Dinamation, Pacific Bell, IBM Corporation, Apple Computer, 
Inc., and Pacific Bell in addition to National Geographic in developing Science 2000. 

Explorations in Middle School Science: The Physical Science Program was initiated by 
Education Systems Corporation, a developer of large scale computer assisted instructional 
programs. This company was later acquired by Josten’s Learning Corporation. 

The MECC World GeoGraph development project was completed by the Minnesota Educational 
Computing Consortium without assistance from any other firms. MECC, however, is widely 
recognized as one of the most experienced, wide-ranging educational software publishing houses 
with highly professional staff resources and fairly low internal operating costs that resulted in a 
very cost-efficient project and negated the need for outside development assistance or 
investment. 

Adequacy of Resources. In each case the CDE grants provided the margin of support that was 
required to stimulate the development of the four software products that otherwise would never 
have gotten beyond the “wish list” category of instructional materials development. While each 
of the software publishers had sufficient resources to develop the products, all reported that 
market conditions would have kept them from risking the capital to do so without the state’s 
subsidy. 
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VI. Program Support, Resources and Constraints 

1. Budget. Separate budgets were negotiated for each of the software development projects. 
The amounts provided by the CDE and the partnership firms were as follows: 



Provided by Publisher’s 



Publisher 


CDE Grant 


Investment 


Total 


National Geographic Society 


$ 500,000 


$ 1,210,000 


$ 1,710,000 


Mindscape 


14,850 


Unknown 


Unknown 


MECC 


100,000 


190,100 


290,100 


Josten’s Learning Corp. 


167,500 


332,500 


500,000 


C&C Software 


62,000 


154,100 


216,100 


Advanced Ideas 


41,480 


Unknown 


Unknown 


Decision Development Coro. 


884.380 


538.347 


1.423.107 


Totals 


$ 1,770,210 


$ 2,425,047 


$4,139,307 



2. Cost Benefits. As a partnership venture, the software development program was initially 
conceived to be cost effective: the CDE hoped that it would get a good return on its 
investment in terms of technology-based instructional materials to meet important 
curriculum needs and the business firms publishing the products would get a fair return 
on their investments from profits on sales in both the California and other educational 
markets. The state expects to receive royalties on the out of state sales of the products. 
Most if not all, of the states investment in the completed programs is expected to be 
returned by 1996. The cost-benefit analysis in Phase III of this study specifically 
addresses this question. 

3. Leveraging and Institutionalization. It is clear that the CDE was able to stimulate or 
“leverage” more than was originally expended in the program. Well over two million 
dollars was leveraged by the state’s investment of just over one and three-quarters million 
dollars. 
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Evaluation Plan for 1989-1992 



Software Development: The original proposal was expanded to amplify evaluation of the 
software development projects. CETAP has obtained assistance from a senior marketing 
consultant from Regis-McKenna Inc., a marketing research company that specializes in the 
assessment of software products. The following summarizes the evaluation used plan for 
software development: 

Review of Existing Records: An extensive search of existing information located at the CDE and 
at the developer locations was completed. 

Project Staff Interviews: Telephone interviews with available developers of software products 
were conducted to determine answers to questions not fully addressed by the Self-Assessment 
and to solicit additional information. 

Self-Assessment by Developers: The CETAP staff developed a Software Development Project 
Self-Assessment Inventory that was completed by three of the four completed projects and by 
one of the terminated projects. This instrument asks a wide range of questions in the following 
categories: background information, planning, content, product development, evaluation plan, 
project outcomes, funding and support factors, marketing, and recommendations. 

Software User Survey: The CETAP staff developed a survey distributed to educators who have 
and/or are using the software produced by the Software Development Projects. This instrument 
asks a wide range of questions. Items for this survey include: background, staff development, 
curriculum content and technology applications, instructional features, and software product 
evaluation. This survey was administered to a sample of at least 30 users of each of the products 
at beta and non-beta sites. It was completed by 33 users across all projects. Due to the low rate 
of return, data derived from this survey should not be considered conclusive. One of the 
products had only recently been completed, and another was still under development at the time 
of this study. Thus, developers were only able to supply the CETAP project with a few names of 
users. 

On the next page is a matrix which list each of the instruments as Data Sources and shows the 
level of emphasis each instrument focuses on for each of the planned evaluation questions for the 
Software Development Projects. 
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Instrumentation Matrix: 
Software Development 


Data Sources 


To What Extent... 


# = Primary Emphasis 
© = Secondary Emphasis 
O = Little or None 




Self-Assessment 

Inventories 


Interview Protocol 


Teacher Surveys 


Project 

Proposals/Plans 


Project Evaluation 
Reports 


1. Were the program outcomes attained? * 


• 


• 


© 


© 


• 


2. Were major activities implemented as planned? it 


• 


• 


o 


© 


0 


3. Did the project support the state curriculum frameworks? 


• 


• 


• 


• 


• 


4. Was staff development that supported implementation provided? * 


t 


f m 


• 


© 


• 


5. Was project developed coordinated with other programs? it 


• 


• 


© 


© 


© 


6. Did student performance (academic and affective) improve? 


§ 


© 


© 


0 


• 


7. Were there desired changes in instructional practices (teacher perf.)? 


<§ 


© 


• 


0 


© 


8. Was implementation restricted and why? 


© 


© 


o 


• 


© 


9. Was implementation supported or facilitated and why? it 


© 


<§ 


0 


• 


© 


10. Was there dissemination of knowledge and/or products? 


§ 


© 


© 


© 


© 


1 1 . Were unanticipated outcomes assessed? it 




• 


© 


© 


© 


12. Was there equal access for students, teachers, and administrators? 


© 


© 


• 


© 


© 


13. Did the project assist other schools to implement technology? it 


© 


© 


© 


0 


© 


14. Did new business and higher education partnerships result? 


• 


• 


0 


0 


• 


15. Did the program implement the CDE goals and initiatives? 


• 


• 


0 


0 


© 


16. Do teachers and administrators value the program(s)? 


<§ 


© 


• 


o 


• 


17. Can the program be evaluated and show cost benefits? 


% 


© 


© 


0 


• 


18. Should the projects be continued, discontinued, or expanded? it 


% 


• 


0 


0 


• 


19. Was assistance provided by the CDE? it 


• 


• 


o 


• 


• 


20. Should the state continue to fund software development? it 


• 


• 


• 


© 


• 


21 . Have there been educational benefits from software development?^ 


© 


• 


© 


0 


• 


22. Have lessons been learned that could improve future projects?^ 


• 


• 


• 


© 


• 



★ Added to questions required by the RFP 
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The following pages summarize the information reported by the developers of the software 
projects and the findings of the user surveys collected for each project. 

National Geographic Society GTV: 

A Geographic Perspective on American History 



I. Background Information 

GTV is a presentation tool, a large database to be used as a research tool, and a multimedia word- 
and-image processor, designed to be used throughout the year as a supplement to the textbook. 

A standard pattern of GTV usage is that the teacher introduces a lesson to the whole class with 
one or two video features. Students then follow up each lesson with activities by the teacher, 
using computers if and when appropriate. At this point, GTV becomes a workstation in the back 
of the room where small groups of students may research a topic, write their own captions to 
images, and prepare their own shows. An opening GTV lesson can easily fit within the standard 
45-minute period; follow-up activities vary in length of time depending on the extent of the 
assignment the teacher selects. 

The National Geographic Society (NGS) of Washington, D.C. submitted GTV : A Geographic 
Perspective on American History as one of several software development partnership projects 
funded by the CDE and the Legislature between 1987 and 1990. 

II. Planning 

This project was initiated by Apple Computer, NGS, the CDE and a variety of educators who 
were included in project planning. Figure 1 shows the level of involvement of various parties in 
the planning process. The primary planners were curriculum developers and software 
developers. 

Figure 1: Level of Involvement of Various Parties in Planning Process 



Teachers 
Administrators 
Software developers * NGS 
Curriculum developers * NGS 
California Department of Education 
Consultants 
Apple Computer 

An advisory committee was established for this project which consisted of seven educators, four 
media consultants, and a psychologist and which met for two full days in order to provide major 
input into the conceptual design of the product. Individual members of the committee provided 
follow-up consulting in their areas of expertise, with core educational and content reviewers 
reviewing outlines, scripts and rough cuts for each of the four laserdisc sides. 



Level of Involvement 

None Moderate , High 




1 3 3 3 
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The NGS staff involved in this project consisted of a director, an executive producer, a manager 
of educational research and development, an editor and an associate editor. Lucasfilm Learning 
contributed an executive producer and a senior programmer. Apple Multimedia Lab supplied a 
director and the University of Maryland provided a history and geography consultant. 

III. Project Plan 

The project objectives were reported to have been accomplished and activities were completed 
as planned. A major change was in the decision to teach a combination of history and 
geography instead of geography alone, as was originally planned. This modification better 
reflected the reality of the middle school curriculum as well as the California framework, and 
allowed the product to serve a wider audience. 

IV. Content 

A. Curriculum Content 

History-social science, problem-solving and study skills were the major curriculum areas 
targeted by GTV. In designing the software, which was intended to serve grades 5 through 12, 
several curriculum resources were consulted: Model Curriculum Standards, Grades 9-12; Model 
Curriculum Standards, Grades K-8; Technology in the Curriculum Resource Guides; and other 
state curriculum guides and key textbooks. The History-Social Science Curriculum F ramework, 
1988 (draft and final version), was the California framework consulted. 

GTV was designed to provide an introduction and overview to content covered in detail in the 
textbooks. Subject matter ranged from pre-Columbian times to the present. In tailoring the 
program to the California Framework, the three mandates which were followed most closely 
were: Relating History and Geography; Linking Past to Present; and Building Thinking Skills, 
Study Skills, and Participation Skills. 

Potentially, this product could be used to stimulate and increase students’ attention and interest, 
to expand instruction beyond what can normally be provided with textbooks, to introduce and/or 
conclude a lesson, to provide added activities to the existing lesson and to provide extended 
practice or related activities to the existing lesson. The program is appropriate for use by 
individual students, small groups of two to five students, or entire classrooms. 

B. Staff Development 

A teacher should be familiar with very basic computer operations in order to use this software, 
with proficiency in using the program requiring at least a half a day of self-guided practice. 

There is a “user friendly” teachers’ guide and hotline support available, but no training specialist, 
since it was believed that more than anything, teachers need time to work on their own to 
integrate the program into their own curriculum. Also, there are several product demonstration 
resources available to teachers: California Technology Project regional consortia, Model 
Technology Schools, California Hi story- Social Science sites, as well as many key districts and 
California’s two Geography Alliances. 

The company provided staff development funding for teacher training . Fee-based training is 
available through Optical Data Corporation, the sales distributor. The National Geographic 
Society has exceeded the $300,000 teacher training commitment outlined in the four-year GTV 
contract with the State of California. As stipulated in the contract, the Society donated more than 
$100,000 annually prior to the publication of GTV towards activities to improve geographic 
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education. In the two years following the publication of GTV, the Society has donated more than 
$50,000 annually towards GTV training in California. 

C. Documentation 

GTV consists of several teaching tools. There are two double-sided laserdiscs. One consists of 
60 shows including video features, American Journals, and Population Clocks. The second 
consists of 1600 still images so that students and teachers can make their own shows. (There is 
computer software which lets the user: Play Their Own Shows; Search for Images by Keyword; 
Read Their Own Captions; Write Their Own Captions; and Make Their Own Shows.) Finally, 
there are two print reference materials: Physical Map of the United States; and America on 
Parade - a timeline of U.S. History. 

The teacher’s guide includes: Directions on how to hook up the TV, laserdisc player, and 
computer; Tongue-in-cheek step-by-step tour of the computer software; Troubleshooting section; 
Computer basics review; and Quick reference to all computer software features. The Guide also 
includes curriculum specific materials in its description of the scope and sequence of all video 
features. Also, background information and synopses of each show, and follow-up activities to 
do with and without the computer are provided. The Macintosh and IBM versions provide Help 
features. 

D. Product Features 

The table below indicates the degree to which each of the features listed is emphasized in the 
product, as stated by NGS on the Self-Assessment Inventory. 

Figure 2: Features of Product 

Level of Emphasis 

Not a Moderate Major 

Feature Emphasis Emphasis 

Discovery learning 
Drill and practice 

Educational game 
2 Simulation 

*«•= Tutorial 

1 Utility 

£ Integrated learning system 

^ Adaptability 

§* Special needs students 

c Interactive 

Performance-based branching 
h- Testing 

Curriculum framework support 
Ability to save student progress 
Interactive laserdisc 
Multi-media 



Following are some of the main features of GTV according to the National Geographic Society: 

• Discovery learning is implemented through the video features which introduce or 
review a subject. They demand, by their nature, a student response. Their content 
is often point of view, with which students will identify, research the facts and then 
draw their own conclusions. 
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• Utility is provided through a built-in database, keyword search, word processor, and 
video editor for making and customizing shows. 

• Adaptability is ensured because the program can be implemented with or without a 
computer. The teacher is able to choose shows or activities, and can even make 
shows. 

• Special needs are addressed through the program’s demonstrable success with the 
at-risk student. Attention is given to motivation, visual learning, brief captions and 
flexibility. 

• Interactivity is promoted through the interactive tool, database, and image-and-word 
processor. 

• The program was written specifically to The History-Social Science Framework. 

• Ability to gauge student progress is implemented because users can save both 
shows and their location in the sequence of the program. 

• The program is a multi-media product, i.e., it consists of two double-sided 
laserdiscs, computer software, teacher’s guide, and cable to link computer and 
laserdisc. 

• An Apple I1GS, Macintosh or IBM computer is suggested for use with this 
software, along with keyboard, mouse and monitor. A TV monitor and a laserdisc 
player are required, however. 

• No other software packages are needed for use with this product. 



V. Product Development 

According to the developer, teachers were involved extensively in most phases of product 
development (Figure 3). 



Figure 3: Involvement of Teachers in Product Development 

Level of Involvement 

Project/Product Planning Not al all Moderately Extensive ly 

Preparation of proposal for CDE 
Correlation with Frameworks 
Formulation of instructional goals^S^^^ 

Design of Product 

Technology applications 
Teacher modification features 
Features for students with special needs 
Motivational features 
Reinforcement features 
Ideas for lesson plans/support materials 
Pilot Testing and Program EvaL 

Pilot testing of software 
Classroom trials 
Revision of materials after tests 
Staff Development 

Preparation of materials 
Conducting stafl development for users 
Marketing 

Development of strategies 
Contributing to marketing/distribution 
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Once the National Geographic Society and its consultants and Lucasfilm had a good outline of 
the project, a proposal was sent to the CDE. The CDE evaluated each rough cut and made 
suggestions for changes. The CDE also evaluated the Teacher’s Guide outline and drafts and 
helped announce the project’s availability. AB 1470 grants helped certain individual schools 
purchase GTV. The CDE continues to provide support for the product, offering 
recommendations for demonstration sites and uses of teacher training funding and other product- 
related suggestions. 

VI. Evaluation Plan (Beta Testing) 

Two hundred and fifty contacts including twenty-five teachers from nine school districts were 
used to determine need and feasibility for the program. 

The formative evaluation during laserdisc development included approximately three hundred 
students from eight school districts, and twenty-five teachers. Students and teachers were shown 
rough cuts and asked to rate the shows according to their appeal and their appropriateness to 
educational objectives. Suggestions were taken on the kinds of follow-up activities and materials 
preferred. 

Formative evaluation during computer software development included approximately four 
hundred and twenty-five students in five school districts, and four instructional design reviewers 
and classroom observers. The instructional design review was achieved by spot testing the 
functional software “shell” in classrooms. The first pilot test was a month-long alpha trial of the 
Table of Contents Module including the software, “check” laserdisc and a few printed directions. 
The second pilot test was of the Beta Table of Contents Module and the Alpha Showmaker 
Module including multimedia editor, slide database, word processor, “check” laserdisc, and a 
few printed directions. 

Seventeen classrooms averaging twenty-five students each were used in final field testing. 
Approximately one hundred and twenty-five of the students were 7th graders, and three hundred 
of them were eighth graders. Approximately thirty students fall into each of the following 
categories: bilingual, special education, urban, and gifted. Three hundred and ninety-five could 
be considered suburban. The out of state school districts involved included Montgomery 
County, MD and Fairfax County, VA. The California districts included Cupertino, San 
Francisco Unified, and Marin County. Field test sites were selected to ensure a wide range of 
social, cultural, economic and intellectual factors in the student, and a low level of technological 
expertise on the pan of the teachers. Proximity to either Lucasfilm or the National Geographic 
Society to allow visitation was also taken into account. 

For final field testing, evaluators observed teachers working with GTV in a planning period, 
using the product in class, and training other teachers to use it. The evaluators also questioned 
teachers informally during their visits to the schools and probed their answers to a questionnaire 
as a kind of structured interview at the end of the test period. Much information was obtained, as 
well, through initial instruction of the teachers in using the software and accompanying 
equipment. 

VII. Project Outcomes 

In general, the software was found to be easy to use and to have high appeal to students and 
teachers. Despite a typically low level of familiarity with technology, teachers were able to 
learn to use the program quickly although some adjustments were called for. The most 
significant hurdle may be the equipment itself. It is often necessary to purchase special cables or 



O 



15 19 



Software Development Partnership Program 



adapters to connect the laserdisc player to the present array of monitors or TVs in the school. 
Problems encountered in obtaining these cables, which cost under $40.00 each, are less a matter 
of money than the purchasing process at the schools. Except for a laserdisc player lent by 
Pioneer, all necessary hardware was lent to the schools by the project. 

The overall appeal of the product to teachers lay in its ease of use and in the content of the longer 
segments. In contrast, the shorter segments were viewed as lacking enough detail to substitute 
for a lesson, and there was some concern about the high cost of the technology in relation to its 
information yield, a perception which was not tested after the addition of the database and the 
Presentation Editor modules. Students were attracted to the technology itself. Opinions varied 
on the length of segments, music, narrators. Black-and-white images received low marks, and 
students preferred the moving images to the stills. 

According to the developer, GTV specifically helped students learn traditional material in a vivid 
new form by practicing reading and writing within the context of “making television.” Teachers 
were provided with a program which can be customized to their own particular needs, consisting 
of software and laserdiscs. 



Figure 4: Increases in Teacher Abilities Brought About by GTV 



Level of Increase 
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Not at all Moderate Significant 

Expand resources beyond textbook 
Develop new concepts/understanding 
Encourage problem solving/critical thinking 
Support student-centered learning 
Make teaching more interesting 
Provide interactive experiences 
Provide simulation experiences 
Gain confidence in use of technology 
Integrate technology with curriculum 
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There were systematic efforts to determine the cost-benefits of the software development project 
in comparison with conventional school textbook development programs. However, GTV is 
intended to illuminate, not cover, textbook material. The cost for a class of thirty-five would be 
about $28 per student which is equivalent to the cost of a textbook before volume discounting. 

Among the unanticipated outcomes of the project was the discovery of the importance of high- 
quality production values to the students. Also unexpected was the enthusiasm with which “at 
risk” students greeted this supplement to reading and writing. These students showed increased 
motivation, interest, and self-esteem. 

GTV will be upgraded to make the product compatible with computeroperating system changes. 

VIII. User Survey Results 

A. General Information 

Fifty-seven surveys were sent to a variety of educators using a mailing list provided by National 
Geographic. Fourteen (25%) of the surveys were returned by educators from a variety of 
settings, ranging from rural to urban, and small to large schools. Five were teachers (including 
one special education teacher and one department chair), four were coordinators or directors of 
special projects, one was a principal, one a vice principal, and one a computer lab technician. 
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Twenty-one percent of the educators were at schools which had received AB 1470 School-Based 
Educational Technology Grants. 

All respondents stated that GTV was used in teaching U.S. History or social science. Students in 
all grades from kindergarten to adult were using the product at the sites surveyed, but the greatest 
concentration was in 8th grade. 

A majority of the respondents (71%) were aware that the development of GTV was funded in 
part by the CDE. 

Advertisements (43%) and professional conferences (36%) were the most commonly cited 
sources of initial information about GTV. Educators also learned about the product from sales 
representatives, county office media/technology consultants, listings in technology newsletters 
(such as Computer-Using Educators, CUE), and a variety of other sources. 

More than one-third of the educators stated that a special project funding source had purchased 
GTV. 

Most respondents (86%) decided to try GTV because it related to a curriculum need of the 
students. Many also cited looking for ways to increase student interest (79%), a desire to use 
technology programs in class (64%), student desire for technology programs (64%), students 
getting more out of technology programs (64%), and promotional information that made the 
program sound very effective (64%). Half of the educators were seeking ways to improve 
instruction in history-social science and many (43%) had heard from others that the program 
facilitated student understanding of social science concepts. 

B. Staff Development 

Half of the respondents reported that prior computer use was necessary to use GTV. Just over 
half (57%) stated that they had received staff development in the use of the product. This 
training was most often conducted by the California Technology Project Regional Consortia 
(38%), National Geographic staff (38%), school site staff (38%), and district staff (38%). CUE 
and outside consultants also provided staff development on GTV. Half of the educators 
receiving staff development indicated it was adequate. 

Those reporting adequate staff development described it as a brief overview of the prepackaged 
teaching modules with use of the laser disk player and teacher’s guide, a brief explanation of 
customizing lessons, and a brief hands-on session with the materials. 

Those who reported that staff development was inadequate cited the brevity of the presentation 
and insufficient step-by-step instructions in the use of the product. When asked what staff 
development they thought was necessary to use GTV, teachers suggested a demonstration of 
product capabilities, hands-on time, step-by-step development of a lesson, demonstrations of 
model lessons, and tutoring with customized lesson development. Two teachers suggested 
including a tutorial or demonstration program. 

Close to half (42%) of the respondents stated that extensive hands-on training was essential for 
teachers to effectively integrate GTV into the instructional program. Knowledge of the 
frameworks (21%), cooperative learning training (21%), and hardware in the classroom (14%) 
were also cited as essential for effective integration of GTV into the instructional program. 
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C. Curriculum Content and Technology Applications 

A variety of U.S. History topics in the framework were listed by respondents as being covered by 
GTV. Among the specific history topics listed were: Pre-Columbian Days to Revolution, 
Independence to Civil War, Centennial to Twentieth Century, and World War I to Present. 
Overall, the US History topics received an average rating of 3.6 on a scale of one to five for 
adequacy of coverage. US Geography was also listed by several respondents; it received an 
average rating of 3.8. 

Three of the teachers (21%) have the computer and laserdisc player required to use GTV in their 
classroom. Most (86%) have equipment at their schools that teachers can share. Only one stated 
that their school could not afford the equipment needed to use GTV. About two-thirds reported 
that the required equipment is easy for teachers to use. 

Eight respondents (57%) felt that HyperCard or other multimedia products could be used to 
increase the effectiveness of GTV. 

D. Instructional Features 

As shown in Figure 5, GTV was reported by users to be very effective in promoting student 
interest. The program was also found to be flexible enough to be used with small or large groups 
and students with special needs. Its interactivity and ability to reinforce other instruction were 
also highly rated. 

Figure 5. Instructional Features 



Components can be modified by teachers 
Useful for individuals and small or large groups 
eg Can be used by students with special needs 
|5 Can be used by students interactively 

g Promotes student interest 

Provides problem solving opportunities 
Helps students that are behind to "catch-up" 
Reinforces other instruction 
Includes tests to assess student progress 



Average Rating 

Not a Moderately Very 

Feature Effective Effective 
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E. Software Product Evaluation 

All of the educators stated that they use GTV as a classroom presentation tool. Most also use it 
for students in small group settings (79%) and with individual students (71%). Thirty-five 
percent use it with large groups of students. 

Most respondents (64%) have used less than half of the entire curriculum package. Twenty-nine 
percent have used more than half but less than 80 percent of GTV. 

Two respondents use GTV on a weekly basis. Both use it twice per week. Eight of the educators 
surveyed use GTV on a monthly basis, with an average of 3.8 uses per month. An average of 3.2 
teachers use the program at each site surveyed. Most GTV users surveyed teach history-social 
science (86%) but a few teach English-language arts (14%). 
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The following figures show the average ratings given to various evaluation criteria. The 
evaluation criteria were based on those used by the California Instructional Video and California 
Computer Software Clearinghouses. 
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Figure 6. Effectiveness of Instructional Design 

Average Rating 

Unacceptable Good Exemplary 

Content follows a logical development students can follow 
Provides students with positive, appropriate responses! 

Students are actively engaged in the program | 

Stimulates critical thinking and problem solving I 
Presentation of material is better than textbook on subject ! 

Suitable for both individual and group use 
Can be used effectively without teacher direction 
Multiple paths provided for individual students to meet objective! 

Permits simple, frame-by-frame access for students who need it | 

Beginners helped by well-developed, preprogrammed sequences 
Innovative approaches encourage creativity by users 
Encourages users to locate and use additional learning resources 
Help screens provide clear, understandable explanations 
Pace is appropriate for the grade level of students 
Graphic cues support instruction and help navigate through program 
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Figure 7. Effectiveness of Interactive Features 

Average Rating 

Unacceptable Good Exemplary 

Students can easily and independently operate the program 
Indexing is thorough, simple to use, and accurate 
Menus or icons allow students direct access to parts of program 
Allows students to change responses before they are processe 
Teacher or student can determine pace 
Lesson sequences branch based on student responses 
Changes difficulty of content based on student responses 
Students can go back through program to review 
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Figure 8. Support of Curriculum Frameworks 

Average Rating 

Unacceptable Good Exemplary 

Content supports specific curriculum objectives in Frameworl 
Content supports instructional theme(s) in Framework 
Content integrates instructional themes in Framework! 

There are curriculum-related sequences ready for teacher use! 
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Figure 9. Student Interest Level 



Average Rating 

Unacceptable Goo d Exemplary 



Interest level of content is well suited to the students 
Content and format stimulate student curiosity and imagination 
Students eager to use program repeatedly and share it with friends 
Provides for various learning modalities (visual, auditory, etc.) 
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As shown in Figure 6, the most effective elements of GTV’s instructional design were its logical 
development of concepts (which is easy for students to follow) and its ability to present material 
better than the history-social science textbook. The most highly rated interactive features were 
the program’s pace can be set by the student or teacher and that students can go back through the 
program to review (see Figure 7). According to the educators surveyed, GTV was not able to 
change the difficulty of its content based on student responses. GTV’s support of the California 
Curriculum Framework for History-Social Science was found to be exemplary by almost all of 
the educators surveyed (see Figure 8). As shown in Figure 9, high ratings were also given to the 
program’s ability to stimulate student interest in history-social science. The average overall 
rating of the instructional value of GTV was 4.4 (between good and superior on a five point 
scale). 

Only three respondents were aware of products similar to GTV. Two listed Optical Data 
Corporation science laserdiscs and one mentioned HyperStudio combined with laserdiscs. 
Neither of these products specifically supports history-social science instruction. 

There was no consensus among teachers as to wether GTV or traditional printed materials were 
more cost-effective. The prices that the schools or districts would be willing to pay for GTV 
ranged from $300 to $900, with the average being $579. In general, teachers reported the list 
price of $995 was too high. California schools were able to purchase the program for $650 until 
December 1, 1991. 

F. User Comments 

A teacher from Project TOPS (a Level II Academic-Technology Model Project) wrote in: 

“We have highly recommended this program to teachers who have visited 
us. Students even demonstrated it at the National Education Association 
(NEA) conference in San Francisco Iasi March,” 

IX. Funding and Support Factors 

A. Funding Support 

Several organizations that contributed both direct payment and in-kind support were very 
important to the success of the project. Lucasfilm Learning provided video production, software 
design and programming services. Apple Computer was involved in early planning, funding, 
and software technical assistance. The CDE provided major funding as well as consultation on 
curriculum . The National Geographic Society Educational Program provided teacher training 
and public information. Also very important was the advance purchase of many units by Optical 
Data Corporation, which serves as the distributor. Of significant, though slightly less, 
importance was the in-kind support from IBM which provided technical advice and equipment 
support; the direct and indirect support from the National Geographic Society Educational 
Foundation, which provided fundraising; the direct payment of funds from Pacific Telesis and 
from Iowa Carver Trust for equipment and training; and the in-kind support of the San Francisco 
Schools which provided its early endorsement, test sites, and system-wide application of the 
program. In-kind support was also provided by the Pioneer Communications of America, Inc., 
the Kern County High School District, and the Iowa Department of Education. 

The National Geographic Society contributed $1,210,000 to the original version of GTV. This 
investment, as well as the investments of its partners in the project was recovered because the 
company was able to write off certain prototype phases. It would not have been possible for the 
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National Geographic Society to have developed and produced this product without the $500,000 
CDE Software Development Project Grant. 

This grant and the National Geographic Society’s original budget did not cover the cost of future 
computer versions beyond the IIGS version. Since then, there has been a Macintosh version 
developed with financial and in-kind assistance from Apple Computer and an IBM version 
developed with in-kind assistance from IBM. Although GTV has an outside distributor, the 
National Geographic Society incurs expenses in the hundreds of thousands of dollars each year in 
supporting and maintaining the product line. This was not part of the original budget or CDE 
grant. 

B. Supporting and Impeding Factors 

Funding Factors. The overall funding available for the project and the amount of funding 
provided by the CDE greatly supported its planning, implementation and outcomes. The 
schedule of CDE payments was also very helpful. 

Product Development Factors. The establishment of working relationships with development 
partners and the performance of those partners were reported to be the two most important 
factors affecting the planning, implementation and outcomes of the project. Other important 
factors included the matching of the program’s content with the California History-Social 
Science Framework for marketing purposes both within and outside of California, matching the 
technology applications to teachers’ skills, and providing staff development materials for product 
users. 

Evaluation and Monitoring Factors. Making design changes based on internal evaluation of 
the project, assessing the capabilities of teachers to use the program effectively, and field testing 
the software were the three factors believed to be most helpful to the development of the 
software. Of slightly less importance were making design changes based on external 
evaluations, and evaluating the effects of the product on student learning and attitudes. 

Clarifying content objectives and making design modifications at the request of the CDE project 
monitor helped the final product somewhat, although the ongoing interaction with the project 
monitor was reported to have slowed down product development to some extent. 

X. Marketing 

Optical Data Corporation of Warren, N.J. is the sole distributor of GTV. The company is a 
leading producer and distributor of laserdiscs, with a field staff of regional sales managers, a 
customer support service for ordering and information, a technical support hotline, and an 
inservice program for users of laserdiscs. Customers may either order or preview kits through 
this distributor. 

Both NGS and Optical Data Corporation advertise and promote GTV. NGS sponsored the 
original direct mail brochure and produced three short informational videos - one is available 
free of charge to schools; two are for use in presentations. GTV is now offered in both the NGS’s 
Educational Technology Catalog and in Optical Data Corporation’s annual catalog and other 
product brochures. GTV is exhibited at most major educational conventions; GTV workshops are 
scheduled at many of them. Optical Data Corporation features GTV in its promotional newsletter 
to the schools. The product has attracted the attention of both TV and print media. GTV is 
marketed in California as it is elsewhere. California, however, has received the benefit of a four- 
year teacher training commitment as part of its contract with NGS, leading to greater product 
seeding in key demonstration sites, more teacher training workshops and ten half-day hands-on 
workshop/demon stration s . 
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The suggested retail price of GTV is currently $995.00. In its first year of publication, GTV was 
offered at a special introductory price of $650.00 (until 12/1/90). In place of a discount, 
California educators were provided with free staff development, valued at $300,000. 

GTV sales have exceeded the expectations of the developer. It has just passed the break-even 
point on development costs. There is considerable cost involved, however, in maintaining the 
product line- marketing, services, royalties, teacher training assistance. To date, California 
represents 29% of total sales units. The state has received $22,163 in royalties from the out of 
state sales of GTV. 

XI. Use of Product by School-Based Grant Projects 

Several of the AB 1470 School-Based Educational Technology Grant Projects used a portion of 
their funding to purchase GTV. In fact, GTV was the third most popular software product, used 
by 36 projects (14% of those surveyed). In general, it received high ratings for effectiveness 
averaging 4.3 on a 5 point scale. A teacher surveyed at one of these projects submitted the 
following description of a student who was helped by the product: 



Guz joined our student population in March. He was an ESL student from East 
Los Angeles whose father was relocated to Northern California. A very shy boy, 
Guz was slow to integrate into our community, until he became involved in 
technology. Guz utilized the GTV laserdisc to produce a video segment about 
immigration. 

I did not have time to give him the instruction other students received earlier in 
the year. He asked to borrow the manuals and taught himself. His presentation 
was the best in the class. His esteem rose tremendously. 

I received a card from Guz at the end of the year in which he described this as the 
besi year he ever_ ha<L 



GTV was also mentioned specifically by several students who were surveyed at School-Based 
Grant projects. These students found the program to be both useful and fun to work with. 
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I. Background Information 

The MECC Corporation of Minneapolis submitted World GeoGraph as one of several software 
development partnership projects funded by the CDE and the Legislature between 1987 and 
1990. 

II. Planning 

This project was initiated by MECC and a variety of educators were included in project planning. 
Figure 1 shows the level of involvement of various parties in the planning process. . 

Figure 1: Level of Involvement of Various Parties in Planning Process 



Teachers 
Administrators 
Software developers - MECC 
Curriculum developers 
California Department of Education 
Consultants 



Level of Involvement 




In order to evaluate the software, suggest support materials, and help review the project 
generally, an advisory committee was established which consisted of four professionals in 
education. Three of these taught at the college level and specialized either in teacher training or 
in geography. The fourth was a teacher of social studies in the Minnesota public school system. 

The MECC employees most involved in the development of the project included two software 
designers, three programmers, and two graphic designers. 

in. Project Plan 

The project objectives were accomplished and activities were completed as planned. No major 
changes in the scope of work, timelines, etc. were negotiated before the projected was completed. 

IV. Content 

A. Curriculum Content 

Geography and history-social science were the major curriculum areas targeted by MECC World 
GeoGraph. In designing the software, which was intended to serve grades six through adult, 
both state-adopted curriculum materials and the History-Social Science Framework for 
California Public Schools, K-12, 1988, were consulted. 

The three topics of the California Framework which were covered thoroughly in the design of the 
software were geographic literacy, physical geography and world regional geography. Critical 
thinking skills, nationalism in the contemporary world, hemispheric relationships in the post war 
era, contemporary issues in the world today, and international economic concepts were given 
strong but slightly less emphasis, and cultural literacy was only lightly touched upon. 
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World GeoGraph is a discovery-learning product with a wide variety of uses in the classroom. 
Students can use it to freely explore a nation and/or topic in geography or to compare nations. It 
includes a wealth of demographic and economic information about nations and regions. 

Teachers can use it to illustrate or demonstrate geographic concepts, international trade, global 
political issues, macroeconomics, and demographics. It can be used with a high degree of 
teacher guidance or none at all. While its use by individual students is ideal, it can be used 
nearly as effectively by small or large groups also. Potentially, the program can be used to 
stimulate and increase students’ attention and interest, to expand instruction beyond what can 
normally be provided with textbooks, to provide added activities to the existing lesson, and to 
provide extended practice or activity related to the existing lesson. 

B. Staff Development 

A teacher who is familiar with basic computer operations could become proficient in the use of 
this software in one or two hours without any special training. The World GeoGraph Classroom 
Guide and the World GeoGraph User’s Guide provide all necessary instructional information. 

C. Product Features 



Figure 2 shows the level of emphasis given to the product’s various features. 



Figure 2: Features of Product 
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Level of Emphasis 

Not a Moderate Major 

Feature Emphasis Emphasis 

Discovery learning 
Drill and practice 
Educational game 
Simulation 
Tutorial 
Utility 

Integrated learning system 
Adaptability 
Special needs students 
Interactive 
Performance-based branching 
Testing 

Curriculum framework support 
Ability to save student progress 
Interactive laserdisc 
Multi-media 




The most important features of the product were discovery learning, utility, and the ability to 
save student progress. The discovery learning feature provides a non-linear, open-ended tool that 
students can use to explore and analyze topics of their own choosing, allowing them to draw their 
own conclusions from the data. As a utility, the program serves as a “geography processor” or 
database and map-making tool that allows for printing and saving of reports. Students can save 
their work for later resumption or teacher examination. 

This software requires the use of an Apple IIGS computer with at least 768K and any RGB color 
monitor. LCD panels and video projectors would be useful but not required. No other software 
packages or equipment are necessary to accompany World GeoGraph. 
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V. Product Development 



As shown in Figure 3, teachers were involved in many phases of product development. 
Figure 3: Involvement of Teachers in Product Development 



Project/Product Planning 

Preparation of proposal for CDE 
Correlation with Frameworks 
Formulation of instructional goals 
Design of Product 

Technology applications 
Teacher modification features 
Features for students with special needs 
Motivational features 
Reinforcement features 
Ideas for lesson plans/support materials 
Pilot Testing and Program Eval. 

Pilot testing of software 
Classroom trials 
Revision of materials after tests 
Staff Development 

Preparation of materials 
Conducting staff development for user: 
Marketing 

Development of strategies 
Contributing to marketing/distribution 



Level of Involvement 

Not at all Moderately Extensively 




VI. Evaluation Plan (Beta Testing) 

Four classrooms in two different school districts were used to evaluate World GeoGraph in terms 
of clarity, appropriateness, ease of use, and usefulness, and to consider any changes that might be 
made to the program to improve it in these and other areas. 

The evaluation began with the four teachers being consulted and shown how to use the product. 
The teachers then used the product with their students for four weeks. The product designers got 
input from the teachers during this period and observed the product being used in class and 
interviewed both students and teachers. 

Surveys of both students and teachers were used in the quantitative evaluation process. 
Qualitatively, classroom observations, interviews with both teachers and students along with 
their incidental comments, and teacher assessments of students’ work all contributed to the 
evaluation. 

There were approximately twenty-five 6th graders and twenty-five 7th graders involved in the 
final classroom field test. The teachers had all necessary hardware in their classrooms, and the 
schools had all necessary hardware for teachers to share. MECC gave memory chips to some of 
the participating schools. There is little information on the field testing of this program since no 
report was prepared. 
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VII. Project Outcomes 

MECC World GeoGraph specifically helped students learn to use a database as a tool for inquiry 
and analysis in geography, and to observe possible correlations among various types of data. 
They were better able to observe patterns of similarities and differences among nations and to 
understand basic geographic concepts of location, place, and relationships within places and 
regions. 

Teachers were able to enhance classroom lectures by using the program as a tool to display 
maps, chans, and data. They were also able to create student handouts and examine geographic 
data in preparation for classroom activities. Figure 4 shows project staff’s assessment of the 
product’s impact on teaching ability. 

Figure 4: Increases in Teacher Abilities Brought About by World GeoGraph 

Level of Increase 

Not at all Moderate Significant 

Expand resources beyond textbook 
Develop new concepts/understanding 
S Encourage problem solving/critical thinking 
3 Support student-centered learning 

_ Make teaching more interesting 

■2 Provide interactive experiences 

8 Gain confidence in use of technology 
*“ Integrate technology with curriculum 

1 2 3 4 5 

There were no systematic efforts to determine the cost-benefits of this software development 
project in comparison with conventional school textbook development programs, and there were 
no unanticipated outcomes of the project. While the program will have to be revised every two 
or three years, MECC plans to provide periodic data updates through information disks as long 
as the demand continues. 




VIII. User Survey Findings 

A. General Information 

Using a list of users provided by MECC, thirty surveys were sent to a variety of educators using 
a list of users provided by MECC. Seven (23%) of the surveys were returned - four by 
California educators and three by educators in other states. Four were teachers (including one 
department chair), one was a director of computer instruction, and two were technology 
coordinators. One teacher was at a school which had received an AB 1470 School-Based 
Educational Technology Grant. 

Two teachers used World GeoGraph to teach social studies, one used it for geography, one used 
it for both, and one used it to support mathematics instruction. The remaining two teachers did 
not actually use the product with students due to technical difficulties - one said it was too slow 
and too expensive to purchase a copy for each computer and the other didn’t have enough 
memory for the program to run. The educators surveyed used World GeoGraph with middle or 
high school students ranging from grade six to twelve. 

Advertisements (57%) were the most commonly cited source of initial information about World 
GeoGraph. Educators also learned about the product from listings in technology newsletters 
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(such as Computer-Using Educators), articles in professional journals, district curriculum 
coordinators, at conferences, and from fellow staff members. 

Three educators stated that a teacher had purchased World GeoGraph. The other copies were 
purchased by a school district, a special project, a school, and a computer coordinator. The most 
common reasons for trying the program in class (cited by four respondents each) were: it related 
to a curriculum need of the students, the teacher likes using technology programs, and the 
students like using technology programs. Three stated they had heard that World GeoGraph 
helps students to understand geography and social science concepts. Other reasons listed were: 
students get more out of technology-assisted instruction, teachers look for ways to increase 
student interest, and that it was highly recommended by a colleague. 

B. Staff Development 

All of the respondents reported prior computer use was necessary to use World GeoGraph. One 
educator received staff development in the use of the program from staff at the school site and 
from MECC. This teacher stated that the staff development was not adequate because it was too 
brief and did not provide enough hands-on experience. 

When asked what staff development they thought was necessary to use World GeoGraph, one 
stated that reading the manual was sufficient, two indicated basic knowledge of computer 
operation was necessary, and two reported in-depth hands-on training was needed for all aspects 
of the program. 

Two teachers stated little or no additional training was needed to integrate the World GeoGraph 
into the instructional program. One stated sharing ideas with other teachers was important, one 
indicated training in the California Curriculum Frameworks was necessary, and another stated 
training in database operation was needed. 

C. Curriculum Content and Technology Applications 

Three specific topics were listed by educators as being covered by the program: world 
geography, world demographic data, and world social/economic data. Both teachers who listed 
world geography gave World GeoGraph a 5 on a scale of 1 to 5 for adequacy of coverage. The 
teacher who listed demographic and social/economic data gave these topics a 3. 

Most of the educators (71%) have the Apple IIGS needed to run World GeoGraph in their 
classrooms. The remaining teachers have access to shared equipment at their schools. Only one 
of the seven educators reported that the equipment was easy for most teachers to use. 

Four of the respondents (57%) indicated word processing/desktop publishing software could be 
used with World GeoGraph to effectively promote learning. Three stated that multimedia 
software would be an effective complement to the program and another three stated that graphics 
software was very effective. Three also reported that database management software worked 
well to promote student learning. One teacher reported that database software was not effective 
with students. 

D. Instructional Features 

As shown in Figure 5 below, World GeoGraph was found to be very effective in providing 
student problem solving opportunities. The program was also found to be flexible enough to be 
used with small or large groups and students with special needs. Its ability to promote student 
interest and its reinforcement of other instruction were also highly rated. 
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Figure 5. Instructional Features 



Components can be modified by teachers 
Useful for individuals and small or large groups 
vj Can be used by students with special needs 
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Most of the educators surveyed used World GeoGraph in more than one instructional setting. 
Usage was spread evenly between large groups, small groups, individual students, and teacher 
presentation. 

A majority of the respondents (60%) have used more than half but less than 80 percent of the 
entire curriculum package. The remaining 40 percent have used less than half of the program. 

Three respondents use the program on a weekly basis (between one and three uses per week). 
Two of the educators surveyed use World GeoGraph on a monthly basis. Both use it once per 
month. The other two educators do not use the program with students. An average of 1.6 
teachers use the program at each site surveyed. Four respondents use World GeoGraph to teach 
history-social science and one uses it to teach math. 

The following figures show the average ratings given to various evaluation criteria. The 
evaluation criteria were based on those used by the California Instructional Video and California 
Computer Software Clearinghouses. 

Figure 6. Effectiveness of Instructional Design 

Average Rating 

Content follows a logical development students can follow 
Provides students with positive, appropriate responses 
Students are actively engaged in the program 
m Stimulates critical thinking and problem solving 

? Presentation of material is better than textbook on subject 

g Suitable for both individual and group use 

g Can be used effectively without teacher direction 

® Multiple paths provided for individual students to meet objective 
= Permits simple, frame-by-frame access for students who need it 
> Beginners helped by well-developed, preprogrammed sequences 
Innovative approaches encourage creativity by users 
Encourages users to locate and use additional learning resources 
Help screens provide dear, understandable explanations 
Pace is appropriate for the grade level of students 
Graphic cues support instruction and help navigate through program 

1 2 3 4 5 
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Figure 7. Effectiveness of Interactive Features 

Average Rating 

Unacceptable Good Exemplary 

Students can easily and independently operate the program! ” 

Indexing is thorough, simple to use, and accurate! 

Menus or icons allow students direct access to parts of program! 

Allows students to change responses before they are processed 
Teacher or student can determine pace! 

Lesson sequences branch based on student response/ 

Changes difficulty of content based on student responses! 

Students can go back through program to review! 
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Figure 8. Support of Curriculum Frameworks 

Average Rating 

Unacceptable Good Exemplary 

Content supports specific curriculum objectives in Frameworl 
Content supports instructional theme(s) in Framework 
Content integrates instructional themes in Framework 
There are curriculum-related sequences ready for teacher use 




Figure 9. Student Interest Level 
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S Interest level ol content is well suited to the students 

§ Content and format stimulate student curiosity and imagination 
g Students eager to use program repeatedly and share it with friends 
75 Provides for various learning modalities (visual, auditory, etc.) 



Average Rating 

Unacceptable Good Exemplary 




As shown in Figure 2, the most effective elements of World GeoGraph’ s instructional design 
were its ability to be used with small and large groups and its stimulation of critical thinking and 
problem solving skills. The most highly rated interactive features were that the program’s pace 
can be set by the student or teacher and that students can change their responses before they are 
processed (see Figure 3). World GeoGraph's support of the California History-Social Science 
Framework was found to be good to exemplary by the educators from California (see Figure 4). 
As shown in Figure 5, good ratings were also given to the program’s ability to stimulate student 
interest in history-social science. The average overall rating of the instructional value of World 
GeoGraph was 4.75 (between good and superior on a five point scale). 

Three respondents were aware of products similar to World GeoGraph. All three mentioned PC- 
Globe, which is available for Macintosh and IBM-compatible computers rather than the Apple II 
GS. Also mentioned were Geo Quiz and MECC US GeoGraph, a follow-up product funded 
entirely by MECC. 

F. User Comments 

A high school math teacher stated that he successfully used World GeoGraph to illustrate the 
demographics of the Middle East to students during the Gulf War. He said the comparative data 
features of the program were excellent and helped the students to understand where the war was 
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and why it was being fought in addition to presenting the basic concepts of statistical analysis. 
His overall reaction to World GeoGraph was: “ Good job guys!” 

IX. Funding and Support Factors 

A. Funding Support 

The in-kind staff time provided by the University of Minnesota, the University of San Francisco, 
and Bloomington Public Schools was of light-to-moderate importance. The direct payment of 
$100,000 in funds through the CDE’s Software Development Project Grant was essential to the 
project. 

MECC contributed $190,000 to the development of this software, an investment which it has 
succeeded in recovering under the present arrangement. If MECC were to develop and produce 
this software without state support, they estimated that at least $500,000 in sales would be 
necessary in order to justify the risk of developing this product, which has an estimated useful 
life of 4-5 years. 

B. Project Expenditures 

The total expenditures for MECC World GeoGraph were $290,000. All costs of the project were 
covered by the amounts invested by MECC and the CDE. Figure 10 gives a breakdown of the 
expenses of project development as reported on the Self-Assessment Inventory. 



Figure 10: Project Expenditures 




Expenditure Amount 



ill Professional staff 


$160,000 


Q| Support staff 


$10,000 


B Materials/supplies 


$9,000 


^ Contract services/ 


$45,000 


expenses 




CD Staff Benefits 


$30,000 


ITTI Technology hardware 


$45,000 


Total 


$290,100 



C. Supporting and Impeding Factors 

Funding Factors. The overall funding available for the project greatly supported its planning, 
implementation and outcomes. The amount of funding provided by the CDE and the schedule of 
payments both also figured very strongly in supporting this project. 

Product Development Factors. The development of staff training materials for product users 
was believed to be the single most important factor in developing MECC GeoGraph. Of slightly 
less importance were the interaction with the project’s advisory group, the matching of 
technology applications to teachers’ skills and the limitations of hardware in schools. Also of 
slightly less importance was the matching of content with the California Curriculum Frameworks 
for marketing within the state. Considerations for marketing outside of California in general, and 
correlating the software with curriculum in other states also were only slightly supportive. 
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Evaluation and Monitoring Factors. Making design changes based on internal evaluation of 
the project, and evaluating the effects of the software on student learning and attitudes were the 
two factors reported to be most helpful to the development of the software. Of slightly less 
importance were making design changes based on external evaluations, assessing capabilities of 
teachers to use the programs effectively, and field testing of the software. 

X: Marketing 

The product is being distributed through multiple channels, including directly from MECC, via 
educational and retail dealers, and through educational distributors. MECC’s authorized 
representatives also sell World GeoGraph, including some in California. There are ads in 
educational journals and direct-mail brochures have been sent out. The product is also included 
in most educational computing catalogues. 

According to MECC, the original suggested retail price was $139, including the Classroom 
Guide. The California school price was to be $89 when obtained directly from MECC at cost. 
The sales price has steadily declined and is presently $41.40 for California schools. 

Sales are meeting expectations both within and outside of California. No exact sales figures 
broken down by state are available. From 6/90 to 3/91, the product has sold approximately 2,400 
units (approximately $135,000). The projection for 1992 is approximately 2,500 units 
($150,000). The product is now on the “down side” of the sales cycle. Sales in fiscal year 1990 
were approximately 5,000 units ($300,000). Sales have been comparable or perhaps a little 
better than those of competing educational products. 

To date, the state has received $30,904 in royalties from out of state sales of MECC World 
GeoGraph. 

XI. Use of Product by School-Based Grant Projects 

Five AB 1470 School-Based Educational Technology Grant Projects used a portion of their 
funding to purchase World GeoGraph . PC Globe, a competing product, was purchased by six 
projects. World GeoGraph’s average rating for effectiveness was 4.0 on a five-point scale. 

Sixth grade students, when asked about World GeoGraph at one project site, replied: 



“It gives a choice. We can select what it is we want to explore." and “It 
helps me in learning; I can find out find out for myself the information, in 
the way I want to find it, without having to be programmed to do it in a 
certain way.” 



Students were able to use the program to easily compare detailed demographic and economic 
data between different countries and to make predictions based on the comparisons. 
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I. Background Information 

The Decision Development Corporation of San Ramon, California submitted Science 2000 as 
one of several software development partnership projects funded by the CDE and the Legislature 
between 1987 and 1990. 

II. Planning 

This project was initiated by the Decision Development Corporation and the CDE. Figure 1 
shows the level of involvement of various parties in the planning process. 

Figure 1: Level of Involvement of Various Parties in Planning Process 



Teachers 
Administrators 
Software developers - DDC 
Curriculum developers 
California Department of Education 
Consultants 



In order to evaluate the software, suggest support materials, and help review the project 
generally, an advisory committee was established by the CDE which consisted of seven 
professionals in education and two corporate advisors. 

III. Project Plan 

The project objectives were accomplished and activities were completed as planned. The 
software and the curriculum were both revised and expanded in scope. 

IV. Content 

A. Curriculum Content 

Among the curriculum resources consulted in the design of the software, which was targeted for 
use by 7th graders, were Model Curriculum Standards, Grades K-8; Technology in the 
Curriculum Resource Guides ; Subject Matter Project Documents ; and Project 2061, Science for 
all Americans. Science Framework for California Public Schools, 1990, was the Curriculum 
Framework used. 

Science 2000 is a multimedia product designed for use by teachers using a constructivist, 
cooperative learning model in a one-year science curriculum. It can be used to stimulate and 
increase students’ attention and interest, to expand instruction beyond what can normally be 
provided with textbooks, to actually replace textbooks and to support teachers. 



Level of Involvement 

None Moderate High 




1 2 3 4 5 
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B. Staff Development 

While a teacher needs no prior experience with computers or software in order to use the 
program, training is necessary, and one or two days’ time would be needed in order to become 
proficient. A videotape providing an overview of the program, and a teacher’s guide providing 
all necessary instructional information, are available. " " 

The company provides staff development for the schools independently and in collaboration with 
education agencies. Product demonstration sites will be available to teachers at up to forty pilot 
schools in California and at IBM staff development sites. Schools will be able to pay for a brief 
staff development orientation session from the company. In addition, dealers are providing staff 
development. 

C. Documentation 



Besides the teacher guide and the overview videotape, one year of 7th grade teachers’ science 
lesson plans and support materials such as student investigations, software activities, two double- 
sided laserdiscs and four kits are available. 

D. Product Features 



Figure 2 shows the level of emphasis given to the product’s various features. 



Figure 2: Features of Product 
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Interactive 
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Ability to save student progress 
interactive laserdisc 
Multi-media 




The most important features of the product were discovery learning, which provides 
opportunities for students to problem solve and explore; utility, which uses software as a tool for 
editing text, worksheets and spreadsheets; adaptability, which allows teacher modification; 
Curriculum Framework support; interactive laserdisc; multimedia components including 
software, laserdiscs, kits, video and text; student assessment items on or offline; teacher tool for 
curriculum/lesson planning; and modification capabilities for report preparation by students or 
teachers. Simulation and interactive capabilities are also very important. 
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This software requires the use of a Macintosh or IBM PC compatible computer, and a Pioneer 
LDV 2200 or 4200 laserdisc player. While no other software packages are required, graphics, 
reference, telecommunications, and networking software are suggested. 

V. Product Development 

As shown in Figure 3, teachers were involved extensively in most phases of product 
development. 



Figure 3: Involvement of Teachers in Product Development 
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Thirteen of the seventeen consultants involved in the development and testing of Science 2000 
were teachers divided almost equally between content designers and content reviewers. The 
remaining four consultants served as writers. Collaborative activities of the project developers 
and the CDE’s Office of Educational Technology consisted of bimonthly meetings. 

VI. Evaluation Plan (Beta Testing) 

Twenty-four seventh grade teachers in twelve school districts were consulted to determine 
teacher acceptance of Science 2000 according to appropriateness of its content and the ease of 
use of its software, laserdiscs and kits. Fifteen hundred of the students were urban; 1500 of the 
students were suburban; and 200 of the students were rural. Pre-test and post-test models were 
used formatively and summatively including student and teacher surveys, journals and classroom 
observations. Decision Development Corporation bought some hardware components for some 
sites, although most schools had all the hardware needed. 

VII Project Outcomes 

The project was still in progress at the time of the self-assessment, so no outcomes or evaluation 
results are available at this time. 
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Science 2000 is expected to need updating for new information every two or three years. 

VIII. User Survey Findings 

A. General Information 

Since Science 2000 was still in development at the time of this survey, beta test sites had to 
be surveyed rather than regular users. Six surveys were returned by educators who are 
currently testing the program with their classes. All were teachers (including a department chair, 
a mentor teacher, and the director of TASC II, a Level II Academic-Technology Model Project).’ 
One third of the educators were at schools which had received AB 1470 School-Based 
Educational Technology Grants. 

All respondents stated that Science 2000 was used in teaching science in 6th or 7th grade. At 
TASC II, Science 2000 was tested in the project’s Knowledge Lab. 

Two thirds of the respondents were aware that the development of Science 2000 was funded in 
part by the CDE. 

District media/technology specialists (50%) were the most commonly cited sources of initial 
information about Science 2000. Educators also learned about the product from a school 
principal, a school department head, articles in professional journals, the CDE, and a district 
curriculum coordinator. 

Four respondents (67%) decided to test Science 2000 with their classes because they enjoy using 
technology programs in class and were looking for ways to increase student interest in science. 
Also cited were fulfilling curriculum needs, students getting more out of instruction when 
technology is used, recommendations from colleagues, student enjoyment of technology 
programs, a principal or department head’s desire to try the program, and a variety of other 
reasons. 

B. Staff Development 

All survey respondents reported prior computer use was necessary to use Science 2000. Two 
thirds stated they had received staff development in the use of the product. This training was 
conducted by school staff, district office staff, and Decision Development Corporation (DDC). 
Half of the educators who received staff development indicated it was adequate. 

Those who reported the staff development as adequate described it as on-going consultation with 
DDC regarding specific software and curriculum issues, one-on-one instruction, and provision of 
equipment, cables, and disks. 

When asked what staff development they thought was necessary to use Science 2000 effectively, 
teachers suggested basic training in the use of the Macintosh or IBM computer and laserdisc 
system, a broad overview of the program’s clusters and how they fit together, practice both with 
and without students, and classroom management and cooperative learning training. 

To effectively integrate Science 2000 with the curriculum, teachers reported general computer 
knowledge was essential, new classroom management techniques are needed (to rotate 30-35 
students through one or two workstations), adequate preparation time must be provided, and a 
method for assessing student progress and assigning grades must be developed. 
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C. Curriculum Content and Technology Applications 

A wide variety of physical and life science topics were listed by teachers as being covered in 
Science 2000, including: Ecology, Pesticides, Food Webs, Food Chains, Habitats, Topographic 
Maps, Geology, The Water Cycle, The Environment, Microbiology, The Solar System, and 
Photosynthesis. The beta versions of these units were all given fours and fives on a scale of one 
to five rating completeness of coverage. The average rating was 4.5. 

Teachers recommended several pieces of additional equipment to be used with the program in 
addition to the required Macintosh or IBM compatible computer and laserdisc player. 
Recommendations included: video equipment, LCD projection panels (many Macintosh screens 
are too small for group instruction), CD-ROM drives, probes for experiments, and modems. 

All but one of the six teachers surveyed have the computer and laserdisc player required to use 
Science 2000 in their classrooms. Two have equipment at their schools that teachers can share. 
Two stated that their school could not afford the equipment needed to use the program (some 
equipment was loaned to teachers by DDC for beta-testing purposes. All six reported the 
required equipment is not easy for teachers to use. 

Two teachers used CD-ROM databases with Science 2000 and found them to be effective, two 
used telecommunications programs, and one used Logowriter and other Apple lie software. 
HyperCard, graphics, and database software were also found to be effective when used in 
conjunction with Science 2000. 

D. Instructional Features 

As shown in the Figure 4, the most effective features of Science 2000 were its ability to be used 
with groups of differing sizes, the problem solving opportunities it provides, and its interactivity. 
Weak areas in the beta version were that it didn’t allow students that were behind to “catch-up” 
easily and that it did not provide an effective means to assess student progress. 

Figure 4. Instructional Features 



Components can be modified by teachers 
Useful for individuals and small or large groups 
<g Can be used by students with special needs 
^ Can be used by students interactively 

2 Promotes student interest 

Provides problem solving opportunities 
Helps students that are behind to ’catch-up’ 

Reinforces other instruction 
Includes tests to assess student progress 

E. Software Product Evaluation 

The teachers surveyed tested Science 2000 in a variety of settings including large and small 
groups, individual student use, and teacher presentation. Because of the difficulty of rotating 
individual students through one workstation, only half of the teachers tried this approach. 

Due to the fact that many features of the program were not operational in the beta version, most 
teachers tested less than half of the entire curriculum package. 



Average Rating 

Not a Moderately Very 

Feature Effective Effective 
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Four respondents used Science 2000 on a daily basis. One teacher used the program constantly, 
all period, for three weeks and another used it for two weeks. At five of the sites Science 2000 
was used by only one teacher. The remaining site stated that five teachers had tried the program 
in a variety of subject areas, including social science, language, math and science. 

The following figures show the average ratings given to various evaluation criteria. The 
evaluation criteria were based on those used by the California Instructional Video and California 
Computer Software Clearinghouses. 



Figure 5. Effectiveness of Instructional Design 
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* Provides students with positive, appropriate responses 
Students are actively engaged in the program 
Stimulates critical thinking and problem solving 
Presentation of material is better than textbook on subject 
Suitable for both individual and group use 
* Can be used effectively without teacher direction 
Multiple paths provided for individual students to meet objective 
Permits simple, frame-by-frame access for students who need it 
Beginners helped by well-developed, preprogrammed sequences 
Innovative approaches encourage creativity by users 
Encourages users to locate and use additional learning resources 
Help screens provide clear, understandable explanations 
Pace is appropriate for the grade level of students 
Graphic cues support instruction and help navigate through program 







FFF ! 1 




gagg" 




















■ 





Science 2000 was designed to be a teacher’s tool - it was not specifically designed to give 
‘right’ answers to students. 

Figure 6. Effectiveness of Interactive Features 
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Students can easily and independently operate the program 
Indexing is thorough, simple to use, and accurate 
Menus or icons allow students direct access to parts of program 
Allows students to change responses before they are processed 
Teacher or student can determine pace 
Lesson sequences branch based on student responses 
Changes difficulty of content based on student responses 
Students can go back through program to review 
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Figure 7. Support of Curriculum Frameworks 

Average Rating 

Unacceptable Good Exemplary 

o Content supports specific curriculum objectives in Frameworl 

S Content supports instructional theme(s) in Framework 

§ Content integrates instructional themes in Framework 

There are curriculum-related sequences ready for teacher use 
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Figure 8. Student Interest Level 
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S Interest level of content is well suited to the students 

§ Content and format stimulate student curiosity and imagination 
g Students eager to use program repeatedly and share it with friends 
73 Provides for various learning modalities (visual, auditory, etc.) 



As shown in Figure 5, the most effective elements of Science 2000’s instructional design were its 
stimulation of critical thinking and problem solving and its flexibility in allowing individual 
students to meet objectives in different ways. The most highly rated interactive features were 
that the program’s pace can be set by the student or teacher and that students can go back through 
the program to review (see Figure 6). Science 2000’s support of the California Curriculum 
Framework for science was found to be exemplary by almost all of the educators surveyed (see 
Figure 7). As shown in Figure 8, high ratings were also given to the program’s ability to 
stimulate student interest in science. The average overall rating of the instructional value of 
Science 2000 was 3.8 (between fair and good on a five point scale). Some teachers were hesitant 
to rate the program since they had not seen the final version. 

Only one teacher was aware of any similar products. He listed Rediscover Science from 
Edunetics and programs from MECC and Jostens. 

Since Science 2000 was beta tested rather than purchased and the price and features of the final 
version had yet to be determined, no teachers were able to evaluate its cost effectiveness. The 
project list price is projected to be $5,000 per unit (including a site software license) with a 25 
percent discount for California schools. 

F. User Comments 

The director of TASC II stated: 



Average Rating 




“Learning as / saw it happen could not have taken place without Science 
2000 " 



She went on to state that student participation and “tum-on” was wonderful while using the 
program. 

Another teacher stated: 



"the short video clips and and slides which we accessed through the 
laserdisc system provided excellent visualsjor students 



These teachers suggested that the software component of Science 2000 would be more effective 
if at least four workstations were available in the classroom instead of just one. 



IX. Funding and Support Factors 

A. Funding Support 

The CDE’s Software Development Project Grant of $884,000 covered the majority of the 
development costs of this project. Partners in the project contributed in-kind support estimated 
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at $538,347. Besides Decision Development Corporation these partners included the National 
Geographic Society, Dinamation, Pacific Bell, the IBM Corporation, and Apple, Inc. 

Both the funding and the intellectual support provided by the state were necessary for Decision 
Development Corporation to have attempted this project. 

B. Project Expenditures 

Since the project was not yet completed at the time of the self-assessment, no final budget 
information is available. 

C. Supporting and Impeding Factors 

Funding Factors. The overall funding available for this development project, the amount of 
funding provided by the CDE, and the schedule of payments all were believed to have greatly 
supported the planning, implementation and outcomes of this project. 

Product Development Factors. Also of the highest importance were the interaction with the 
project advisory group, establishing working relationships with development partners, protecting 
intellectual property of publisher and partners, performance of development partners, and 
matching product content with California Curriculum Framework for marketing both within and 
without California. 

Evaluation and Monitoring Factors. The factors most conducive to the development of 
Science 2000 included establishing project review/evaluation procedures with the CDE, 
clarifying content objectives with the CDE project monitor, ongoing communication with the 
CDE project monitor, and making changes in the product design specifications suggested by the 
internal and external evaluations and by the CDE project monitor. 

X. Marketing 

The product will have a list price of $5,000 per unit (including a site software license), with a 25 
percent discount to California schools. 

California will receive a 10 percent royalty on all out of state sales of Science 2000. 



XII. Use of Product by School-Based Grant Projects 

No projects are currently using Science 2000 as it was not yet on the market. 



Jostens Explorations in Middle School Science: The 
Physical Science Program 



I. Background Information 

Jostens Learning Corporation of San Diego, California submitted Explorations in Middle School 
Science as one of several software development partnership projects funded by the CDE and the 
Legislature between 1987 and 1990. 

II. Planning 

This project was initiated and planned by Jostens, the CDE, and a variety of educators and 
consultants. Figure 1 shows the level of involvement of various parties in the planning process. 
The primary planners were curriculum developers and software developers. 

Figure 1: Level of Involvement of Various Parties in Planning Process 



Teachers 
Administrators 
Software developers * Jostens 
Curriculum developers - Jostens 
California Department of Education 
Consultants 



An advisory committee was established for this project which consisted of six science education 
consultants who developed the basic curriculum strands and the concepts in each strand. 

The Jostens staff involved in this project consisted of a director, a head writer, a project manager, 
a technology coordinator, and two writers. 

III. Project Plan 

The project objectives were reported to have been accomplished and all activities were 
completed as planned. No major changes were made in the scope of work or timelines. 

IV. Content 

A. Curriculum Content 

Science was the major curriculum area targeted by Explorations. In designing the software, 
which was intended to serve grades 6 through 9, several curriculum resources were consulted: 
Model Curriculum Standards, Grades 9-12; Model Curriculum Standards, Grades K-8; 
Technology in the Curriculum Resource Guides, Subject Matter Project Documents ; and other 
state curriculum guides. The California Science Framework, 1978 and the 1984 addendum, were 
the curriculum frameworks consulted. 




Level of Involvement 



1 2 3 4 5 
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The specific framework topics covered were: Matter, Mechanics, Energy Sources and 
Transformations, Heat, Light, Electricity and Magnetism, and Sound. All were covered 
completely except for Heat and Sound, which received moderate coverage. 

Explorations lessons were designed to take up one class period each, but two periods usually are 
required. A total of 30 lessons are included in the program, requiring 30 to 60 class periods to 
complete. Potentially, this product could be used to stimulate and increase students’ attention 
and interest, to expand instruction beyond what can normally be provided with textbooks, to 
introduce and/or conclude a lesson, to provide added activities to the existing lesson and to 
provide extended practice or related activities to the existing lesson. The program is appropriate 
for use by individual students or small groups of two to five students. 

B. Staff Development 

A teacher should be familiar with very basic computer operations in order to use this software, 
with proficiency in using the program requiring at least 40 hours of training by a specialist. 

Sixty hours of teacher training by a Jostens Educational Services Consultant are included in the 
purchase price of the software. In addition, demonstration sites are available in Sacramento, Los 
Angeles, and Irvine. 

C. Documentation 



A teacher’s guide which includes lesson plans, technical documentation, and curriculum specific 
materials is provided with the program. 

D. Product Features 



Figure 2 below indicates the degree to which each of the features listed is emphasized in the 
product. 



Figure 2: Features of Product 
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Level of Emphasis 

Not a Moderate Major 

Feature Emphasis Emphasis 

Discovery learning 
Drill and practice 
Educational game 
Simulation 
Tutorial 
Utility 

Integrated learning system 
Adaptability 
Special needs students 
Interactive 
Performance-based branching 
Testing 

Curriculum framework support 
Ability to save student progress 
Interactive laserdisc 
Multi-media 
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Following are some of the major features of the program: 

• The program provides students with many opportunities to learn and explore on 
their own. 

• All lessons are built around powerful simulations. 

• An on-line notebook/writing processor and calculator are available for student use. 

• The Jostens Integrated Learning System (ILS) which provides a networked 
environment capable of recording each student’s progress is a part of the program. 

• All lessons force the student to interact. 

• All lessons are in-depth studies of concepts emphasized in the framework. 

• An Apple IIGS, Macintosh or IBM compatible computer with a color monitor is 
required. In addition, network hardware is needed, including: a file server with a 
CD-ROM drive, network boards, and cables. IBM compatible computers require a 
voice board. 

• Word processing and networking software is required; an electronic encyclopedia 
program is suggested. 

V. Product Development 

As shown in Figure 3, teachers were involved extensively in most phases of product 
development. All Jostens curriculum writers and staff developers are credentialed and 
experienced classroom teachers. 



Not at all Moderately Extensively 



Figure 3: Involvement of Teachers in Product Development 

Level of Involvement 

Project/Product Planning 

Preparation of proposal for CDE 
Correlation with Frameworks 
Formulation of instructional goals 
Design of Product 

Technology applications 
Teacher modification features 
Features for students with special needs 
Motivational features 
Reinforcement features 
Ideas tor lesson plans/support materials 2 
Pilot Testing and Program Eval. 

Pilot testing of software 
Classroom trials 
Revision of materials after tests 
Staff Development 

Preparation of materials 
Conducting staff development for users 
Marketing 

Development of strategies 
Contributing to marketing/d istributioi 




The CDE collaborated with Jostens during the development process through meetings and 
review of unit and lesson design. 
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VI. Evaluation Plan (Beta Testing) 

As pan of its formative evaluation. Explorations was reviewed by 15 teachers who attended the 
August 1989 Summer Science Institute sponsored by UC Irvine. A networked Apple Ilgs 
laboratory was established at UC Irvine through the assistance of Jostens staff and it was used in 
the evaluation of the program. The major formative evaluation questions were: 

• Do the activities comply with the highest standards of excellence in educational software as 
defined in the Guidelines for Educational Software in California Schools? 

• Are the activities at the appropriate level and interesting? Is the scientific content accurate? 
Are they motivating? Are they up-to-date? 

• How do the activities work when students work individually? In pairs? In larger groups? 

• Do the activities meet teachers’ needs? Do they fit into the teacher’s classroom? 

• Do the on-line tests provide sufficient assessment of individual student progress? Do they 
address teachers’ classroom management needs? 

• Is the documentation satisfactory? 

• What recommendations do the teachers have for improving the materials? 

Teachers were asked to review several of the program’s lessons and then fill out questionnaires 
and rating sheets based on these evaluation questions. The software was also reviewed for 
compliance with state requirements for instructional materials and assessed for consistency with 
the California Science Framework and Guidelines. 

The summative evaluation consisted of five major components: 

• A questionnaire was administered to teachers who were using the materials at the beta test 
sites. 

• Telephone interviews were conducted with teachers and administrators at participating 
sites. Based on these interviews and on feedback from Jostens Education Specialists, one 
site was selected for a detailed case study. 

• Site visits, with interviews of all involved staff, were made to each beta test site. 

• Cognitive outcomes were measured using on-line quizzes and standardized achievement 
measures. 

• The impact of the materials on students’ attitudes towards science were assessed. 

Among the quantitative instruments used in the evaluation were: proficiency tests, student and 
teacher surveys, records of computer use, and school climate surveys. Qualitative instruments 
included: portfolio assessments, student and teacher interviews, teacher assessment of student 
work, classroom observations, teacher journals, and incidental comments by students and staff. 

The alpha and beta testing included 1,150 students at two junior high schools, one in Sacramento 
and one in Los Angeles. Many of the students were bilingual and a fair number were in special 
education or gifted programs. The field test sites were chosen based on student population and 
the availability of equipment. During alpha testing, an adult objective observer used each lesson 
with a small group of students as soon as it was developed. Both the student and the observer 
would fill out reports and these observations were incorporated into the program. The alpha 
testing led to many small but significant changes in the screen designs and the beta testing led to 
a shifting of resources to teacher training and the implementation of strategies and tools. 
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VII. Project Outcomes 

Explorations allowed students to construct a conceptual understanding of ideas outside of their 
normal experiences, and it provided the opportunity to manipulate variables, measure responses, 
and draw conclusions easily. Teachers were free to be the facilitator of learning and not the 
lecturer and were allowed to delve into concepts beyond the understanding of the students (see 
Figure 4). 

Figure 4: Increases in Teacher Abilities Brought About by Explorations in Science 

Level of Increase 

Expand resources beyond textbook 
w Develop new concepts/understanding 
S Encourage problem solving/critical thinking 
H Support student-centered learning 

Make teaching more interesting 
■g Provide interactive experiences 

2 Provide simulation experiences 

Gain confidence in use of technology 
Integrate technology with curriculum 

1 2 3 4 5 




There were no systematic efforts to determine the cost-benefits of the software development 
project in comparison with conventional school textbook development programs. 

Among the unanticipated outcomes of the project was the discovery of a need for greater teacher 
training with a specific need for whole solutions (full curriculum) to make the greatest use of the 
computer. 

Jostens releases at least one upgrade yearly with enhancements. Due to school demand, all 
lessons have been ported to the new Macintosh LC computer. 

VIII. User Survey Results 

A. General Information 

Twenty-six surveys were sent to a variety of educators using a mailing list of users provided by 
Jostens. Six surveys (21%) were returned by educators who are currently using the program with 
their classes. All were science teachers including one who was a science department chair. 

None of the educators taught at schools which had received AB 1470 School-Based Educational 
Technology Grants. 

All respondents stated that Explorations was used in teaching science in 7th and 8th grade. 

One third of the respondents were aware that the development of Explorations was funded in Dart 
by the CDE. 

School principals (67%) and sales representatives from Jostens (33%) were the most commonly 
cited sources of initial information about Explorations. Educators also learned about the product 
from a school library/media specialist and a fellow teacher. 

Four teachers stated that the program was purchased by their district. The other two did not 
know who had made the purchasing decision. 
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Four respondents (67%) decided to try Explorations with their classes because it related to a 
curriculum need of their students. Also cited were student enjoyment of technology programs, 
looking for ways to increase student interest in science, a principal or department head’s desire to 
try the program, information about the program making it look good, teacher preference for using 
technology programs in class, use of the program being requested by the district or department, 
and a variety of other reasons. One teacher stated that she was able to preview the program 
before using it. 

B. Staff Development 

None of the respondents reported that prior computer use was necessary to use Explorations. 

Two thirds stated they had received staff development in the use of the product. This training 
was conducted for the most part by Jostens, with some assistance from school and district staff. 
All of the educators who had received staff development indicated it was adequate. 

The staff development was described as a brief introduction to the computer and the basic 
features of the program. Teachers were advised to practice with the program on their own. The 
built-in tutorial section made it easy for them to learn to use Explorations effectively with a 
minimum of outside support. One teacher stated: “The staff development was good for the 
small amount of time allowed.” 

When asked what staff development they thought was necessary to use Explorations effectively, 
two teachers suggested curriculum integration training, two suggested hands-on time with the 
program, and one stated the none was needed other than learning to use a mouse. One school 
used a networked system managed by a technology specialist which alleviated the need for 
teachers to know anything other than how to coach the students at each individual station. 

To effectively integrate Explorations with the curriculum, teachers felt that information was 
needed on the specific curriculum content of each unit, and that time was needed for them to run 
through each lesson on the computer to see where it belongs in the curriculum. 

C. Curriculum Content and Technology Applications 

A variety of physical science topics were listed by teachers as being covered in Explorations, 
including: Electricity, Energy, Mechanics, and Light and Sound. These units were all given 
fours or fives on a scale of one to five rating completeness of coverage. The average rating was 
4.4. One teacher was disappointed that no earth science topics were included in the program. 

Only one of the six teachers surveyed had the IBM compatible computer required to use 
Explorations in their classroom. The others have equipment at their schools that teachers can 
share. One stated that their school could not afford the equipment needed to use the program. 
Four respondents felt that the required equipment is easy for teachers to use and one stated that 
the equipment is cumbersome and difficult for most teachers to use. 

Word processing and Integrated Learning Systems programs were used effectively in 
conjunction with Explorations by three of the teachers. One of these teachers also used graphics 
software. 

D. Instructional Features 

As shown in Figure 5, the most effective features of Explorations were its ability to be used by 
students interactively, its stimulation of student problem solving, and its ability to assess student 
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progress. Components of the program can not be modified by teachers and it was not found to be 
very effective for students with special needs. 

Figure 5. Instructional Features 

Average Rating 

Not a Moderately Very 

Feature Effective E ffective 

Components can be modified by teachers 
Useful for individuals and small or large groups 
eg Can be used by students with special needs 
5 Can be used by students interactively 

2 Promotes student interest 

Provides problem solving opportunities 
Helps students that are behind to 'catch-up' 

Reinforces other instruction 
Includes tests to assess student progress 

1 2 3 4 5 

E. Software Product Evaluation 




All of the teachers surveyed used Explorations with individual students. In addition, one teacher 
used it with large groups and another used it with small groups. 

Most teachers have used less than half of the entire curriculum package. 

Five respondents use Explorations once per week, and one uses it twice per month. The number 
of teachers using the program at each site ranged from two to 26. In addition to science, the 
program is used in math, English-language arts, and history-social science. 

The following figures show the average ratings given to various evaluation criteria. The 
evaluation criteria were based on those used by the California Instructional Video and California 
Computer Software Clearinghouses. 
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Figure 6. Effectiveness of Instructional Design 



Content follows a logical development students can follow 
Provides students with positive, appropriate responses 
Students are actively engaged in the program 
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Presentation of material is better than textbook on subject 
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Can be used effectively without teacher direction 
Multiple paths provided for individual students to meet objective 
Permits simple, frame-by-frame access for students who need it 
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Graphic cues support instruction and help navigate through program 

1 2 3 4 5 








47 



Software Development Partnership Program 



m 

*c 

<u 

*n 

o 

c 

o 

33 

m 

a 

CT3 

> 



Figure 7. Effectiveness of Interactive Features 

Average Rating 

Unacceptable Good Exemplary 

Students can easily and independently operate the program I 
Indexing is thorough, simple to use, and accurate! 

Menus or icons allow students direct access to parts of program! 

Allows students to change responses before they are processe 
Teacher or student can determine pace! 

Lesson sequences branch based on student responses 
Changes difficulty of content based on student responses! 

Students can go back through program to review! 
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Figure 8. Support of Curriculum Frameworks 

Average Rating 

Unacceptable Good Exemplary 

Content supports specific curriculum objectives in Frameworl 
- Content supports instructional theme(s) in Framework 
Content integrates instructional themes in Framework 
There are curriculum-related sequences ready for teacher use 




Figure 9. Student Interest Level 
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Interest level of content is well suited to the students 
Content and format stimulate student curiosity and imagination 
Students eager to use program repeatedly and share it with friends 
Provides for various learning modalities (visual, auditory, etc.) 



Average Rating 

Unacce ptable Good Exemplary 




As shown in Figure 6, the most effective elements of Explorations’ instructional design were its 
ability to be used without teacher direction, its logical sequence of instruction, and its graphics 
cues which help students navigate through the program. The most highly rated interactive 
features were the menus and icons which guide students through the program and its simple and 
accurate indexing system (see Figure 7). Explorations’ support of the California Science 
Framework was found to be good by almost all of the educators surveyed (see Figure 8). As 
shown in Figure 9, fairly good ratings were also given to the program’s ability to stimulate 
student interest in science. The average overall rating of the instructional value of Explorations 
was 4.2 (between good and superior on a five point scale). 

IX. Funding and Support Factors 

A. Funding Support 

UC Irvine contributed significant evaluation support to the project and UCLA and McDonnell 
Douglas provided moderate support in developing lesson content. 

Jostens contributed $332,000 to the development of Explorations. It would not have been 
possible for Jostens to have developed and produced this product without the $167,500 CDE 
Software Development Project Grant. At the present rate of sales, Jostens will need three more 
years before it recovers its initial investment. The company stated that to support a new product 
development, they would need to sell enough to make $20 for every dollar invested. 
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B. Project Expenditures 

The total expenditures for Jostens Explorations in Science were $500,000. All costs of the 
project were covered by the amounts invested by Jostens and the CDE. Figure 10 gives a 
breakdown of the expenses of project development, as reported in the Self-Assessment 
Inventory. 



Figure 10: Project Expenditures 
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Expenditure 


Amount 


PH Professional staff 


$160,000 


^ Support staff 


$10,000 


| Materials/supplies 


$9,000 


^ Contract services/ 
expenses 


$45,000 


O Staff Benefits 


$30,000 


ITU Technology hardware 


$45,000 


Total | 


$290,100 | 



C. Supporting and Impeding Factors 

Funding Factors. The amount of funding provided by the CDE grant greatly supported the 
planning, implementation and outcomes of the project. The schedule of CDE grant fund 
payments and the overall level of funding were also helpful. 

Product Development Factors. The factors which most supported product development were: 
interaction with the project advisory group, establishment of a working relation with the 
development partners, and protection of the intellectual property of the publisher and partners. 
The performance of the partners also facilitated development. The most severe impediments to 
development were the limitations of hardware in schools and the matching of product technology 
applications to teacher skills. 

Evaluation and Monitoring Factors. The factors which most supported evaluation and 
revision on the product were the assessment of teacher capabilities to use the program effectively 
and evaluation of the effects of the product on student learning and attitudes. 

X. Marketing 

Jostens sales representatives market Explorations directly to the schools. There are no sales 
through dealers or distributors. 

Explorations in Science costs approximately $6,000 per individual station. This amount includes 
a computer, laserdisc player, and software. Quantity discounts are available: the cost for five 
stations is $20,000; for ten stations, $27,500. California schools receive a discount of 20 percent 
off catalog price. 

To date, the state has received $51,500 from the 10 percent royalty on out of state sales. 
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There are 16 Explorations in Science labs currently operating in California with approximately 
32 stations per lab. An additional 84 labs exist in schools outside of California. Sales have been 
much less than predicted in the grant proposal. 

XI. Use of Product by School-Based Grant Projects 

No AB 1470 School-Based Educational Technology Grant Projects reported using Explorations 
in Science. Most School-Based Grants were not large enough to purchase such expensive 
software and the networked computer systems required to use it. 



Summary of Evaluation Findings 



I. Background Information 

Seven software development partnership projects were funded by the CDE between 1987 and 
1990. Self-Assessment Inventories were returned by four completed projects ( Explorations in 
Science, GTV, MECC World GeoGraph, and Science 2000) and one cancelled project (C&C 
Exploring Matter). Surveys were returned by users of GTV, World GeoGraph, beta-test versions 
of Science 2000, and Explorations in Science. 

II. Planning 

The projects were all initiated by the CDE in collaboration with the software developers partners. 
CDE involvement in the planning of the projects was varied - Science 2000 and Explorations in 
Science reported extensive involvement, and the other three projects that completed inventories 
reported only moderate involvement. Teachers were involved in the planning of all projects 
except Science 2000. In general, school administrators had little involvement in project 
planning. All of the projects except Exploring Matter had active advisory committees, 
established either by the developers, or in the case of Science 2000, by the CDE. Educators and 
variety of other individuals, ranging from corporate advisors to psychologists, served on these 
committees. 

III. Project Plan 

The four complete projects stated that, with few modifications, the originally planned activities 
had been completed. 

IV. Content 

A. Curriculum Content 

Two of the completed projects targeted science; the other two, history-social science. These 
areas were seen by the state as lacking in quality software and were considered unprofitable by 
the developers. All of the projects based the curricular content of their programs on the CDE’s 
California Curriculum Frameworks, Model Curriculum Standards, and Technology in the 
Curriculum Resource Guides. 

The products were designed to be used with students in a variety of ways to enrich the 
curriculum and increase interest and motivation. GTV and Science 2000 are interactive 
multimedia products, consisting of computer software and laserdiscs, which can be used for 
teacher presentation or interactively by individual students or small groups. MECC World 
GeoGraph is a discovery learning computer program which can be used by students to freely 
explore topics of their choice or as a teacher presentation aid. Explorations in Science can be 
used by individuals or small groups for discovery learning and performing simulated 
experiments and is part of the Jostens Integrated Learning System. 

B. Staff Development 

Most of the products were designed to be user friendly and to require a minimum of technical 
support. A basic knowledge of computer operation was suggested as a prerequisite for operating 
each of the programs. The time required for a teacher to become proficient in operating the 
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programs ranged from one or two hours for World GeoGraph to several days for Explorations in 
Science, which is the most complex of the programs, requiring a file server networked with 
several learning stations. Jostens includes 60 hours of training in the purchase price of 
Explorations in Science. 

C. Documentation 

All products provide instructions for operating the software and extensive curriculum integration 
information. An entire year’s worth of seventh grade science lesson plans and support materials 
are provided with Science 2000, which is comprehensive enough to replace conventional 
textbooks as the primary source of instruction. 

D. Product Features 

All of the products can be used for discovery learning; all are highly interactive, all can be used 
as utilities for student research or projects; all are highly adaptable to different instructional 
situations, including special needs students; and all except for the uncompleted Exploring Matter 
closely support the state curriculum frameworks. Science 2000 and Explorations in Science have 
assessment components. GTV and Explorations in Science have the ability to save student 
progress. 

V. Product Development 

Classroom teachers were heavily involved in the planning, development, and testing of all of the 
products. The CDE and a variety of consultants also assisted in product development. 
Collaborative activities with the CDE included: meetings, evaluation of content and software 
quality, marketing to California schools, staff development planning and implementation, and 
monitoring of project progress. C & C was the only developer reporting dissatisfaction with the 
level of support and collaboration obtained from the CDE. 

VI. Evaluation Plan (Beta Testing) 

All of the software development projects tested their products extensively in real-life classroom 
situations. Hundreds of teachers and thousands of students were consulted in the testing process, 
using both surveys and classroom observations. Student and teacher suggestions were used in 
making changes and adjustments to the products before their final release. 

VII. Project Outcomes 

Information on project outcomes was only available on GTV, Explorations in Science and World 
GeoGraph. These three programs were both reported to significantly increase teacher abilities to 
expand resources beyond those available in the textbook, and to make teaching more interesting. 
Other areas of strong positive impact on teachers were improvements in integrating technology 
with the curriculum and encouraging of problem solving and critical thinking. 

Student outcomes reported for GTV included; traditional material presented in a vivid new form, 
access to a huge database of images, a chance to practice reading and writing with the context of 
“making” television, and video specifically designed for a media-saturated audience. With 
World GeoGraph students learned to use a database as a tool for inquiry and analysis, observe 
patterns of similarities and differences among nations, observe possible correlations among 
various types of data, and understand basic geographic concepts of location, place, relationships 
within places, movement, and regions. Explorations in Science helped students to understand 
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ideas outside of their normal experience and to easily observe the results of changing variables in 
experiments. " ° 

VIII. User Survey Results 

A. General Information 

A total of 33 surveys were returned by users (mostly classroom teachers) of GTV, MECC World 
GeoGraph, Jostens Explorations in Middle School Science, and Science 2000. A little over half 
of the respondents were aware that the development of these programs was funded in part by the 
CDE. The most commonly cited sources of initial information about these programs were 
advertisements and catalogs, fellow teachers, sales representatives from vendors, and school 
principals. Over two-thirds of the educators surveyed decided to try the program with their class 
because it related to a curriculum need of their students. Other common reasons were looking 
for ways to increase student interest, teacher enjoyment of using technology programs in class, 
and student liking the use of technology. 

B. Staff Development 

Over half of the teachers reported prior computer use was necessary to use the software products 
effectively and about half had received staff development in the use of the products. Software 
publishers were the most commonly cited source of training. Fifty-eight percent of those who 
had received staff development felt that it was adequate. 

C. Curriculum Content and Technology Applications 

With few exceptions, teachers stated the curriculum areas targeted by these products were 
covered adequately. Less than half of the teachers have the equipment necessary to use the 
products in their classroom. More often, equipment was available in the school for them to 
share, often making it difficult to provide students with individual, interactive access to the 
programs. A few stated that their school could not afford the required equipment. Less than 
half of those surveyed felt the equipment was easy for most teachers to use. 

D. Instructional Features 

In general, users felt that the instructional features of the programs were effective. The only 
major weaknesses seen were that programs (especially Explorations in Science) could not be 
effectively modified by teachers for different needs, and that tests were not included to assess 
student progress (The final version of Science 2000 allows modification by teachers and 
includes testing). 

E. Software Product Evaluation 

The programs were used most often as teacher presentation tools and by individual students. 
Extensive use was also reported in large and small group settings. Two-thirds of the teachers 
surveyed had used less than half of the entire curriculum package supplied and 30 percent used 
more than half but less than 80 percent. One teacher used the entire package. Forty-six percent 
of the teachers used the program on a monthly basis, 39 percent on a weekly basis, and 15 
percent on a daily basis. 

Due to differences in design approach there was a great deal of variation in scores for specific 
evaluation criteria between the different programs except that all but Explorations in Science 
scored high on curriculum framework support and student interest level. In general, the 
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programs were rated highly for overall instructional value (average of 4.3 on a scale from one to 
five). 

A third of the educators surveyed were aware of other products similar to state-supported product 
they were using. 

There was no real consensus as to whether the software products or conventional textbook 
instruction is more cost effective. The majority of educators stated that they did not know. 

IX. Funding and Support Factors 

A. Funding Support 

All of the projects stated that development would not have been undertaken without support from 
the state. All of the developers, except for DDC, the developer of Science 2000, contributed 
funding of their own far in excess of the amount provided by the state. The National Geographic 
Society and other organizations whose support it solicited made the biggest contribution in 
funding; they spent $1.2 million in addition to the $500,000 contributed by the state while 
developing G7V. In all, over $2.4 million was invested by the developers in exchange for an 
investment of $1.8 million by the CDE. 

B. Project Expenditures 

Detailed breakdowns of expenditures were provided for three of the products, World GeoGraph, 
Explorations in Science and Exploring Matter. With these three, and probably with the others as 
well, the biggest expense was for professional staff involved in designing, developing, and 
marketing the programs. 

C. Supporting and Impeding Factors 

The factors that most supported project implementation were: the amount of funding provided 
by the CDE, overall funding for the projects, field testing of the software, and the results of the 
projects’ (internal) evaluation of their products. The factor that most consistently impeded 
implementation was the limited hardware available at schools. 

X. Marketing 

Marketing information was supplied by MECC, National Geographic, and Jostens. World 
GeoGraph and GTV are marketed through catalogs, brochures, conference exhibits, 
advertisements, and other means. Both have exceeded sales expectations and have passed the 
break-even point on development costs. Explorations in Science is marketed directly by Jostens 
sales representatives. Jostens has not recovered its investment and sales have been much lower 
than predicted. 

XI. Use of Products by School-Based Grant Projects 

GTV is listed by 14 percent of the projects surveyed, and World GeoGraph is used by a few 
projects. No projects stated that they used Explorations in Science and Science 2000 was not 
available at the time the projects were surveyed. Both GTV and World GeoGraph were rated as 
being highly effective in supporting the instructional goals of the projects they were used in. 
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XII. Recommendations 

• School-Based Grant applicants and other schools need to be made more aware of 
the availability of state-developed software which matches the frameworks. 

• Science 2000 and Explorations in Science and the additional hardware and software 
products required to use them effectively are expensive. The state needs to consider 
how to provide more access to state-of-the-art instructional materials for schools 
which have limited budgets. 

• A more uniform evaluation process for software development projects should be 
incorporated which includes guidelines for performance-based assessments of 
student outcomes resulting from the programs. 

• Establish a procedure for tracking the programs after production and distribution to 
assess their use and impact on teaching and learning. 

• Assess the extent to which software development projects increase teacher 
implementation of the California Curriculum Frameworks and student 
understanding of Framework topics. 

• Continue to evaluate the impact and cost-benefits of development projects such as 
Science 2000. 
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Appendix A: 



Software Development Project 
Assessment Instruments 

•Survey Letter 

•Developer Survey Cover Letter 
•Self-Assessment Inventory 
•Software User Survey 



gl FarWest 

LABORATORY 



SAMPLE 

May 6, 1991 

From: John Cradler and Beth Fisher 

To: Science 2000 Beta Test Teachers 

Subject: Evaluation of Science 2000 



Your name was provided to us an educator who is field testing Science 2000. Because of 
your experience with this program, you are being requested to fill out the enclosed 
survey. The information will be used by the California Educational Technology 
Assessment Project (CETAP). 

CETAP is conducted by the Far West Laboratory to determine the impact of programs 
and projects funded by AB 1470, the Educational Technology Act. The findings of this 
survey are very important to this assessment project and will be used by the California 
Department of Education and the California State Legislature to determine future funding 
for educational technology. 

Your prompt attention to completing and returning this survey will be appreciated. 

Please return to John Cradler at the address provided below by JUNE 14, 1991. 

If there are any questions please feel free to call John Cradler at (415) 565-3018 or Beth 
Fisher at (415) 494-2030 



Far West Laboratory For Educational Research and Development 

730 Harrison Street San Francisco, CA 94107-1242 (415)565-3000 FAX (415) 565-3012 




Far West Laboratory For Educational Research and Development 

730 Harrison Street 

San Francisco, CA 94107-1242 

(415)565-3018 FAX (415) 565-3012 



SAMPLE 



, 199_ 



Dear 



As we discussed on the phone, we are in the process of evaluating AB 1470 and its many programs. The 
objective of the evaluation is to obtain information that will help us improve future state funded 
programs, and guide us in the development of state policy and educational technology programs. . 

The evaluation process includes is to have the software developers and users complete an evaluation 
survey. We will then conduct phone interviews with each developer and a small sample of users. We 
will also have a software evaluation team review each product in light of the original project objectives. 

As a participant in the Software Development Partnership Program, we ask that you and your team 
complete the enclosed evaluation form and return it before . 

As I mentioned, we would appreciate your assistance in contacting 30-50 users of your product. We will 
be sending each user a survey and a cover letter explaining the purpose of the survey. In addition, if you 
already have user information that would be helpful with our research, we would appreciate you sharing 
that with us. 

Please feel free to call me at if you have any questions or comments on this project. We 

look forward to talking with you in the coming weeks. Thank you again for your time and assistance. 



Sincerely, 
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Software Development Partnership Project 

SELF-ASSESSMENT INVENTORY 



Far West Laboratory for Educational Research and Development December 1991 

The purpose of this inventory is to provide the California Department of Education (CDE) and the 
Legislature with information about the outcomes of the software development partnership projects 
funded between 1987 and 1990. 

The inventory is designed to follow CDE guidelines for the California Educational Technology 
Assessment Project (CETAP) being conducted by the Far West Laboratory. It should be completed by 
the person who had primary responsibility for developing the software product, with input from others 
who had key roles in the design and production of the products. 

Please compete the survey by and return it to at the address below. If any 

questions or problems arise while completing the inventory please contact . 



SECTION I: Background Information 

Name of Company: 

Address: 

City: State: Zip: 

Final Product Title: 

Person(s) completing this form: 

Telephone: 

Name of Project Director: Phone:. 

Other person(s) who may be contacted: Phone: 



SECTION II: Planning 



i. 



Indicate who initiated the project concept and design (circle Y or N) and the level of involvement of 
each in project planning (circle the appropriate number): 



Initiated? 



a. Teachers Y N 

b. Administrators Y N 

c. Software developers Y N 

d. Curriculum developers Y N 

e. California Department of Education Y N 

f. Consultants Y N 

g. Other (specify): Y N 

h. Other: Y N 



Level of Involvement 



None 




Moderate 




High 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 



2. Was a planning or advisory committee established for the project? 

Yes 

No . 



(circle one) 

1 

2 



If 1 yes, give the names and affiliations of the members and the major activities of the committee: 



3. Identify the key individuals who were involved in developing the project: 



Name 


Title 


Organization Represented 
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SECTION III: Project Plan 



1. To what extent were the project objectives and activities accomplished? (circle one) 

Completed as planned 1 

Partially completed 2 

Modified but completed 3 

Modified and partially completed 4 



2. If numbers 2, 3 or 4 were answered please explain why. State the project objectives, noting if 
there were any changes from the original objectives: 



Describe: 



3. Describe major changes in the scope of work, timelines, etc., that were negotiated before the project 
was completed. 



Describe: 
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SECTION IV: Content 



A. Curriculum Content 



1 . 



Indicate the curriculum areas that were targeted: 

Mathematics 

Science 

History-Social Science 

English-Language Arts 

Other (specify): 



(circle all that apply) 

1 

2 

3 

4 

....5 



2. Indicate the grade levels the program was designed to serve (circle all that apply): 

Pre-K K123456789 10 11 12 Adult 



3. 



Identify the curriculum resources that were consulted in conceptualizing and designing the software: 



California Curriculum Frameworks 

Model Curriculum Standards, Grades 9-12 

Model Curriculum Standards, Grades K-8 

Technology in the Curriculum Resource Guides 

Subject Matter Project Documents 

State adopted curriculum materials 

Other (specify) 



(circle all that apply) 

1 

2 

3 

4 

5 

6 

....7 



4. Give the title(s) and publication date(s) of the California Curriculum Framework(s) that were 
consulted during the development of the program: 



Title 


Date 















5. List the specific California Curriculum Framework topics that were incorporated into the design of 
the software product, and circle the number that indicates the extent to which each topic was 
emphasized in the finished product. 
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6. What is the suggested length of instructional time for teachers to use this product? 

Describe how the product was designed to be used. Include the length of time recommended for 
each student session (in minutes) and the number of sessions needed to complete the program for the 
average student. If a different length or number of sessions is recommended for different grade 
levels, please specify. 



7. List the potential uses of the product: (circle all that apply) 

To stimulate and increase students’ attention and interest 1 

To expand instruction beyond what can normally be provided with textbooks 2 

To use to introduce and/or conclude a lesson 3 

To provide added activities to the existing lesson 4 

To provide extended practice or related activity to the existing lesson 5 

To replace the textbook.. 6 

Other (specify): 7 

8. Is this program appropriate for: (circle all that apply) 

Individual student use 1 

Small group use (2-5 students) 2 

Whole class use 3 

Other (specify): 4 



B. Staff Development 



1. Describe any support materials for students and/or teachers that are available for use with the 
product: 

Describe: 



2. Teacher experience needed to use program: (circle all that apply) 

No prior experience with computers or software needed 1 

Must be familiar with basic computer operations 2 

Specific training required (specify): .... 3 

3. How much time would be needed for a teacher to become proficient in using the product? 

4. Is a specialist needed to provide product training? yes no 



rj r* 

bo 



o 
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5. Are product demonstration sites available to teachers? 
If yes, where are they located? 



6. Does the company provide staff development for the school or district independently or in 
collaboration with education agencies such as county offices? 



7. Are dealers providing staff development for your product? yes no 

8. Is staff development support provided for purchasers of the product? (circle as many as apply) 



Staff development support is not necessary for schools that purchase the product 1 

The teacher’s guide for the product provides all necessary instructional information 2 

A self-administered staff development tutorial is built-in to the product 3 

The publisher’s sales representatives provide staff development 4 

Schools can pay for a brief staff development orientation session from the company 5 

Staff development costs are covered with purchase price of the product 6 

A fairly intensive staff development workshop is available to schools for a fee 7 

Other (fill-in) 8 



C. Documentation 

1. What teaching tools are available for teachers? 



2. What technical documentation is provided? 



3. Are any curriculum specific materials provided? 




6 



67 



D. Product Features 

1. Circle the number that matches the degree to which each of the features listed below is emphasized 
in the product: 



Not a feature or emphasis > Moderately emphasized > Major emphasis 

1 2 3 4 5 



Technology Applications 

a. Discovery Learning (Provides opportunities for students to problem solve 

and explore) 

b. Drill and Practice (Promotes learning through repetitive practice) 

c. Educational Game (Promotes learning through a game format) 

d. Simulation (Simulates experiences generally unavailable, such as lab 

experiments, world travel, and historical events) 

e. Tutorial (Provides direct instruction, provides practice, tests for mastery, etc.) 

f. Utility (Use of software as tool, e.g., text editor, spreadsheet, worksheets, etc.) 

g. Integrated Learning System (Networked computers for teaching from a 

common set of software) 

h. Adaptability (Program can be modified by teacher) 

i. Special Needs (Program can be used with special needs students) 

j. Interactive (Students use program interactively) 

k. Performance-Based Branching (Automatically adjusts level of difficulty 

according to student responses) 

l. Testing (Program has built-in student diagnostic component 

m. Curriculum Framework Support (Supports Ca. Curriculum Frameworks) .... 

n. Ability to Save Student Progress (Program can record where students have 

left off for use in future sessions) 

o. Interactive Laserdisc (Computer interfaced with videodisc) 

p. Multi-Media (list components): .. 

q. Other (describe): . 

s. Other (describe): 

u. Other (describe): 



Level of Emphasis 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



2. For each of the features where a 4 or 5 was circled, describe how the program implements feature. 
Describe: 
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Software Development Partnership Project 

SELF-ASSESSMENT INVENTORY 



Far West Laboratory for Educational Research and Development December 1991 

The purpose of this inventory is to provide the California Department of Education (CDE) and the 
Legislature with information about the outcomes of the software development partnership projects 
funded between 1987 and 1990. 

The inventory is designed to follow CDE guidelines for the California Educational Technology 
Assessment Project (CETAP) being conducted by the Far West Laboratory. It should be completed by 
the person who had primary responsibility for developing the software product, with input from others 
who had key roles in the design and production of the products. 

Please compete the survey by and return it to at the address below. If any 

questions or problems arise while completing the inventory please contact . 



SECTION I: Background Information 

Name of Company: 

Address: 

City: State: Zip: 

Final Product Title: 

Person(s) completing this form: 

Telephone: 

Name of Project Director: Phone:. 

Other person(s) who may be contacted: Phone: 



SECTION II: Planning 



1. Indicate who initiated the project concept and design (circle Y or N) and the level of involvement of 
each in project planning (circle the appropriate number): 



Initiated? 



a. Teachers Y N 

b. Administrators Y N 

c. Software developers Y N 

d. Curriculum developers Y N 

e. California Department of Education Y N 

f. Consultants Y N 

g. Other (specify): Y N 

h. Other: Y N 



Level of Involvement 



None 

1 


2 


Moderate 

3 


4 


High 

5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 



2. Was a planning or advisory committee established for the project? 
Yes 

No 



(circle one) 

1 

2 



If yes, give the names and affiliations of the members and the major activities of the committee: 



3. Identify the key individuals who were involved in developing the project: 



Name 


Title 


Organization Represented 



















































SECTION III: Project Plan 



1. To what extent were the project objectives and activities accomplished? (circle one) 

Completed as planned 1 

Partially completed 2 

Modified but completed 3 

Modified and partially completed 4 



2. If numbers 2, 3 or 4 were answered please explain why. State the project objectives, noting if 
there were any changes from the original objectives: 



Describe: 



3. Describe major changes in the scope of work, timelines, etc., that were negotiated before the project 
was completed. 



Describe: 
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SECTION IV: Content 



A. Curriculum Content 



1 . Indicate the curriculum areas that were targeted: 

(circle all that apply) 



Mathematics 

Science 2 

History- Social Science 3 

English-Language Arts [ ' 4 

Other (specify): ~ 5 



2. Indicate the grade levels the program was designed to serve (circle all that apply): 

Pre-K K123456789 10 11 12 Adult 



3. 



Identify the curriculum resources that were consulted in 

California Curriculum Frameworks 

Model Curriculum Standards, Grades 9-12 

Model Curriculum Standards, Grades K -8 

Technology in the Curriculum Resource Guides 

Subject Matter Project Documents 

State adopted curriculum materials 

Other (specify) 



conceptualizing and designing the software: 
(circle all that apply) 

1 

2 

3 

4 

5 

6 

7 



4. Give the title(s) and publication date(s) of the California Curriculum Framework(s) that were 
consulted during the development of the program: 



Title : 


Date 
















i 



5. List the specific California Curriculum Framework topics that were incorporated into the design of 
the software product, and circle the number that indicates the extent to which each topic was 
emphasized in the finished product. 



Level of Emphasis in Final Product 

Not at all Moderately Completely 





1 


2 


3 


4 


5 




1 


2 


3 


4 


5 




1 


2 


3 


4 


5 




1 


2 


3 


4 


5 




1 


2 


3 


4 


5 ' 




1 


2 


3 


4 


5 




1 


2 


3 


4 


5 




1 


2 


3 


4 


5 




1 


2 


3 


4 


5 




1 


2 


3 


4 


5 
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6. What is the suggested length of instructional time for teachers to use this product? 

Describe how the product was designed to be used. Include the length of time recommended for 
each student session (in minutes) and the number of sessions needed to complete the program for the 
average student. If a different length or number of sessions is recommended for different grade 
levels, please specify. 



7. List the potential uses of the product: (circle all that apply) 

To stimulate and increase students’ attention and interest 1 

To expand instruction beyond what can normally be provided with textbooks 2 

To use to introduce and/or conclude a lesson 3 

To provide added activities to the existing lesson 4 

To provide extended practice or related activity to the existing lesson 5 

To replace the textbook 6 

Other (specify): .... 7 



8. Is this program appropriate for: ' (circle all that apply) 

Individual student use 1 

Small group use (2-5 students) 2 

Whole class use 3 

Other (specify): 4 



B. Staff Development 

1. Describe any support materials for students and/or teachers that are available for use with the 
product: 

Describe: 



2. Teacher experience needed to use program: (circle all that apply) 

No prior experience with computers or software needed 1 

Must be familiar with basic computer operations 2 

Specific training required (specify): .... 3 

3. How much time would be needed for a teacher to become proficient in using the product? 

4. Is a specialist needed to provide product training? yes no 



5. Are product demonstration sites available to teachers? 
If yes, where are they located? 



yes 



no 



6. Does the company provide staff development for the school or district independently or in 
collaboration with education agencies such as county offices? 



7. Are dealers providing staff development for your product? yes no 

8. Is staff development support provided for purchasers of the product? (circle as many as apply) 



Staff development support is not necessary for schools that purchase the product 1 

The teacher’s guide for the product provides all necessary instructional information 2 

A self-administered staff development tutorial is built-in to the p 
The publisher’s sales representatives provide staff development 



Schools can pay for a brief staff development orientation session from the company 

Staff development costs are covered with purchase price of the product 

A fairly intensive staff development workshop is available to schools for a fee . .. 
Other (fill-in) 

C. Documentation 

1. What teaching tools are available for teachers? 



2. What technical documentation is provided? 



3. Are any curriculum specific materials provided? 
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D. Product Features 



1. Circle the number that matches the degree to which each of the features listed below is emphasized 
in the product: 



Not a feature or emphasis > Moderately emphasized > Major emphasis 

1 2 3 4 5 



Technology Applications 

a. Discovery Learning (Provides opportunities for students to problem solve 

and explore) 

b. Drill and Practice (Promotes learning through repetitive practice) 

c. Educational Game (Promotes learning through a game format) 

d. Simulation (Simulates experiences generally unavailable, such as lab 

experiments, world travel, and historical events) 

e. Tutorial (Provides direct instruction, provides practice, tests for mastery, etc.) 

f. Utility (Use of software as tool, e.g., text editor, spreadsheet, worksheets, etc.) 

g. Integrated Learning System (Networked computers for teaching from a 

common set of software) 

h. Adaptability (Program can be modified by teacher) 

i. Special Needs (Program can be used with special needs students) 

j. Interactive (Students use program interactively) 

k. Performance-Based Branching (Automatically adjusts level of difficulty 

according to student responses) 

l. Testing (Program has built-in student diagnostic component 

m. Curriculum Framework Support (Supports Ca. Curriculum Frameworks) .... 

n. Ability to Save Student Progress (Program can record where students have 

left off for use in future sessions) 

o. Interactive Laserdisc (Computer interfaced with videodisc) 

p. Multi-Media (list components): 

q. Other (describe): . 

s. Other (describe): 

u. Other (describe): 



Level of Emphasis 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

....1 2 3 4 5 

....1 2 3 4 5 

1 2 3 4 5 

....1 2 3 4 5 

.... 1 2 3 4 5 

....1 2 3 4 5 

....1 2 3 4 5 

....1 2 3 4 5 

....1 2 3 4 5 

....1 2 3 4 5 

....1 2 3 4 5 

.... 1 2 3 4 5 

....1 2 3 4 5 



2. For each of the features where a 4 or 5 was circled, describe how the program implements feature. 
Describe: " 



3. Identify the type(s) of hardware used with the product (circle R if required, S if suggested, or N if 
not needed): 



a. Computers (systems supported): r S N 

b. Laserdisc player R S N 

c. Audio/video equipment (specify): R S N 

d. LCD panels, video projectors R S N 

e. CD-ROM drive R S N 

f. Science lab equipment (probes, data acquisition interfaces, etc.) R S N 

g. Color Monitor (specify type): r § N 

h. Other (describe): r § N 

1 ' .... R S N 

....R S N 



4. Identify the type(s) of other software packages which can accompany the product (circle R if 
required, S if suggested, or N if not needed). 



a. Word processing R S N 

b. Desktop publishing R S N 

c. HyperCard (text-based programming language) R S N 

d. Graphics programs R S N 

e. Electronic encyclopedia, reference R S N 

f. Database, spreadsheet, other business software RS N 

g. Telecommunications software R S N 

h. Networking software r § N 

i. Other (describe): R S N 

j- R S N 



SECTION V: Product Development 

1. To what extent were teachers involved in the development of the program? 

(circle the number that indicates the extent to which classroom teachers were involved in each task) 



Not at all > Moderately > Extensively 

1 2 3 4 5 

Project/Product Planning 

a. Preparation of project proposal for CDE 1 2 

b. Correlation of content with Curriculum Framework, etc 1 2 

c. Formulation of instructional goals and objectives for program 1 2 

Design of Product (Content, Features, etc.) 

d. Strategies for using technology applications in lessons 1 2 

e. Ways that teacher is able to modify the program 1 2 

f. Applications for students with special needs 1 2 

g. Motivational or interest-enhancing features 1 2 

h. Reinforcement features (how program reinforces other learning) 1 2 

i. Preparation of lesson plans and teacher support materials 1 2 

Pilot Testing and Program Evaluation 

j. Pilot-testing of units or lessons 1 2 

k. Conducting classroom trials of entire program for evaluation 1 2 

l. Revision of classroom materials after pilot tests and evaluation 1 2 

Staff Development 

m. Preparation of staff development and product documentation materials... 1 2 

n. Conducting staff development programs for adopters of product 1 2 

Marketing 

o. Development of product marketing strategies 1 2 

p. Contributing to marketing and distribution of product 1 2 



3 4 
3 4 
3 4 

3 4 
3 4 
3 4 
3 4 
3 4 
3 4 

3 4 
3 4 
3 4 

3 4 
3 4 

3 4 
3 4 



5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 
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2. Complete the table below as follows: 

• List the names and organizations of all outside consultants who were involved with the 
development and testing of the software product. 

• Briefly describe the product development phases or components in which they were involved. 




Describe: 



SECTION VI: Evaluation Plan (Beta Testing) 

1. Briefly summarize the evaluation plan. 

Number of districts involved 

Number of teachers involved 

Number of classrooms used 



List the evaluation questions or purpose: 



Describe the procedure used for both formative and summative evaluation: 



2. What data sources were used in the evaluation? (circle as many as apply): 



Quantitative 

Standardized achievement test (e.g., CTBS) 1 

Proficiency tests/criterion referenced tests 2 

Portfolio assessment (with scoring rubric) 3 

Student surveys 4 

Student attendance 5 

Teacher surveys 6 

Records of computer use (logs) 7 

School climate surveys 8 

Other (describe): 9 

Other: ... 10 

Qualitative 

Portfolio assessment (adapted to the project) 1 

Student interviews 2 

Case studies 3 

Student journals 4 

Teacher assessment of student work 5 

Classroom observations 6 

Teacher interviews 7 

Workshop evaluation 8 

Teacher journals 9 

Changes in school plans 10 

Incidental comments by students/staff 11 

Other (describe): . .. 12 

Other: ... 13 



3. Describe in detail the final classroom field testing of the program. Include who tested the program, 
where it was tested, and how many teachers and students were involved, 

a. School/District(s): 



b. School Contact Person(s): 

c. Student participants (indicate the number for each grade): 

Pre-K K 1 2 3_ 4 5_ 

6 7 8 9 10 11 12 

Adult 

(fill-in the number of students for each type): 

Bilingual Special Ed. Gifted 

Urban Suburban Rural 

4. Were reports of the evaluation prepared? 

Yes 

No 

If yes, summarize findings: 



Please attach copies of pilot -test and evaluation reports. 

Describe the procedures used, what the findings were, and what changes were made as a result of 
the field test: 



Describe the process/criteria used for the selection of field test sites: 



(circle one) 

1 

2 



5 . Indicate the availability of hardware for conducting the evaluation test (circle one): 

Teacher(s) had all necessary hardware in their classroom(s) 1 

School had all necessary hardware for teachers to share 2 

Development project gave all necessary hardware to the school(s) 3 

Project loaned all necessary hardware to the school(s) 4 

Other (describe); ... 5 

6. Please attach copies of any published reviews of the product which have appeared in The Computing 
Teacher, Electronic Learning, or any other magazine. Also include a copy of the any evaluations 
from the California Software Clearinghouse or any other state agencies. 

SECTION VII: Project Outcomes 

1 . Please describe specific benefits of the project for: 

Students: ' — 



Teachers: 



School Programs: 



2. For each of the following categories, rate the observed impact of the program on teachers using the 
software product developed by the project (circle one number for each item below or circle n/a if 
the item does not relate to your project): 

To what extent did the project INCREASE teacher ability to 



a. 


Expand use of resources beyond the textbook 


Not at all 

1 


2 


Moderate 

3 


4 


Significant 

5 


n/a 


b. 


Develop concepts/understanding not otherwise possible 
to teach 


1 


2 


3 


4 


5 


n/a 


c. 


Encourage problem-solving and critical thinking 


1 


2 


3 


4 


5 


n/a 


d. 


Support student-centered learning 


1 


2 


3 


4 


5 


n/a 


e. 


Make teaching more interesting 


1 


2 


3 


4 


5 


n/a 


g- 


Provide interactive experiences for students... 


1 


2 


3 


4 


5 


n/a 


h. 


Provide simulation experiences 




2 


3 


4 


5 


n/a 


i. 


Gain confidence in their own use of technology 


1 


2 


3 


4 


5 


n/a 


j- 


Integrate technology into the curriculum 


1 


2 


3 


4 


5 


n/a 


1. 


Other (describe): 


...1 


2 


3 


4 


5 


n/a 


m. 


Other: 


...1 


2 


3 


4 


5 


n/a 



3. Were there systematic efforts to determine the cost-benefits of the software development project in 
comparison with conventional school textbook development programs? (circle one) 

Yes i 

no “Illlllilllllllllll"!""! 



If yes, briefly describe how the comparisons were made and what the findings were. 



4. Were there any unanticipated outcomes of the project? (circle one) 

Yes 1 

No "I"..". 2 



If yes* briefly describe any outcomes of project implementation, product development, evaluation, 
revision, publication, etc., that were not anticipated. Include information on how the outcomes were 
identified and assessed, how they affected product development, and how problems were resolved. 



5. How often will the software product have to be revised or updated? (circle one) 

Every year 1 

Every two or three years 2 

Every four or five years 3 

Every California Curriculum Framework cycle 4 



Describe your plans for updating the program: 



0 
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SECTION VIII. Funding and Support Factors 

1. What support services or resources were received from formal and informal partnerships in 

developing the software product? (complete the matrix below as follows) 

• List any software developers, institutions of higher education, foundations, or educational agencies 
that provided formal or informal support in the development of your software product. 

• Briefly describe the resource or service provided (e.g., technical assistance, curriculum consulting, 
programming, linking of resources, etc.) 

• Circle the letter that indicates if the type of service or resource involved direct payment (D) or if it 
was contributed or provided in-kind (I). 

• Indicate the importance of the resource or service to the success of the overall project by circling 

the appropriate number as follows: s 



Not Important > Moderately Important > Very Important 

1 2 3 4 5 



Organization/Agency Resource/Service Type Importance 







D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 






D 


I 


1 


2 


3 


4 


5 



2. Enter the sources and amounts of financial support for the software development project. 

Source of Funding Amount 

Received 



CDE Software Development Project Grant 


$ 


Your Organization 




Partners in Project (Fill-in) 




























Other (Fill-in) 
























Totals 


$ 



3. Have your company and project partners recovered the investments in the project? (circle one) 



Yes 1 

No 2 



If no, summarize the return on your investment to date and estimate when and if you expect to make 
a profit. 1 



4. What is the minimum level of sales that it would take for your company to develop and produce the 
software product without state support? 



Describe: 



5. How much was actually spent on the software development project? Enter the sources and the 
amounts expended for each category. 



Project Expenses 



(Round to the Nearest $) 





Explanation of Expenditure 


State 

Grant 


Company 

Funding 


1000 

Professional Staff Salaries 


Project Director FTE = 






Professional Staff FTE= 






Other: 


















Subtotal for 1000 Series 


$ 


$ 


2000 

Classified/Support Staff Salaries 


Clerical SuDDort FTE= 






Other: 
























Subtotal for 2000 Series 


$ 


$ 


3000 


Staff Benefits: 


$ 


$ 


4000 

Materials/Supplies 






































Subtotal for 4000 Series 


$ 


$ 
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(Round to the Nearest $) 







Explanation of Expenditure 


State 

Grant 


Company 

Funding 


5000 

Contract Services/Expenses 


Services 


Consultants 






Duplication/Packaging 


















Expenses 


Facilities 






Travel 












Subtotal for 5000 Series 


$ 


$ 




Indirect Costs: Indirect costs are not a required item and can 
only be entered when a rate has been 
established. Indirect costs computations 
exclude the 6000 category. Rate % 






6000 

Capital Outlay 


c 

Cl) 

e 


Computers 






c 

CL 

'3 

Or 


Other: 






LU 














Subtotal for 6000 Series 


$ 


$ 


TOTALS 


$ 


$ 


GRAND TOTAL 





6. Did the CDE grant and the amount(s) invested by your company and project partners cover all of the 
costs of developing the product? (circle one) 



Yes 1 

No 2 



If no, describe how the scope of the project was changed and/or how the shortfall was made up. 
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9. Supporting and Impeding Factors. Indicate the extent to which each of the following factors affected 
planning, implementation, and the outcomes of the project (circle N/A if not applicable or one number 
for each item in the list below): 





Greatly 


No Effect 


Greatly 


Funding Factors 


Impeded 






-h 




Supported 




Amount of funding provided by CDE grant 




-1 


-2 


-1 


0 


+1 


+2 


+3 


Overall funding available for development project 


.... N/A 


-1 


-2 


-1 


0 


+1 


+2 


+3 


Schedule of CDE grant fund payments 


....N/A 


-1 


-2 


-1 


0 


+1 


+2 


+3 


Product Development Factors 


















Interaction with project advisory group/panel 


.... N/A 


•1 


-2 


-1 


0 


+1 


+2 


+3 


Establishing working relationships with development partners 


....N/A 


•1 


-2 


-1 


0 


+1 


+2 


+3 


Protecting intellectual property of publisher and partners 


.... N/A 


•1 


-2 


-1 


0 


+1 


+2 


+3 


Performance of development partners 




-1 


-2 


-1 


0 


+1 


+2 


+3 


Matching product content with California Curriculum Framework: 


















• For marketing product within California 


... N/A 


•1 


-2 


-1 


0 


+1 


+2 


+3 


• For marketing the product outside of California 


... N/A 


•1 


-2 


-1 


0 


+1 


+2 


+3 


Correlating product with curriculum in other states 


... N/A 


•1 


-2 


-1 


0 


+1 


+2 


+3 


Considerations for marketing product outside of California 


... N/A 


-1 


-2 


-1 


0 


+1 


+2 


+3 


Limitations of hardware in schools 




-1 


•2 


-1 


0 


+1 


+2 


+3 


Matching product technology applications to teachers’ skills 


...N/A 


-1 


•2 


-1 


0 


+1 


+2 


+3 


Developing staff development materials for product users 


... N/A 


-1 


•2 


-1 


0 


+1 


+2 


+3 


Evaluation and Monitoring Factors 


















Establishing project review/evaluation procedures with CDE 


... N/A 


-1 


•2 


-1 


0 


+1 


+2 


+3 


Clarifying content objectives with CDE project monitor 


...N/A 


-1 


•2 


-1 


0 


+1 


+2 


+3 


Field testing software ; 




-1 


•2 


-1 


0 


+1 


+2 


+3 


Ongoing interaction/communication with CDE project monitor 


... N/A 


-1 


•2 


-1 


0 


+1 


+2 


+3 


Getting final approval of product from CDE 


...N/A 


-1 


•2 


-1 


0 


+1 


+2 


+3 


Assessing capabilities of teachers to use the programs effectively .... 


...N/A 


-1 


-2 


-1 


0 


+1 


+2 


+3 


Evaluating effects of product on student learning and attitudes 


...N/A 


-1 


-2 


-1 


0 


+1 


+2 


+3 


Making changes in product design specifications necessitated by: 


















• Results of project (internal) evaluation 


... N/A 


-1 


•2 


-1 


0 


+1 


+2 


+3 


• Recommendations of external evaluation 


... N/A 


-1 


-2 


-1 


0 


+1 


+2 


+3 


• CDE project monitor 




-1 


-2 


-1 


0 


+1 


+2 


+3 



O 

tKJC 
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SECTION IX. Marketing 

1. How is the software product being marketed? 

Describe how the product is being marketed (a) within California and, if different , (b) outside of the 
state; include information on the sales organization(s), distributor(s), advertising and other 
promotional efforts, etc. 



(Attach brochure describing product if available) 

2. How much does the product cost? 



Provide cost breakdowns for different quantities sold to (a) California schools and (b) outside the 
state. 



3. Are sales of the software product meeting the expectations of the developers? 



Describe briefly how well the product is selling (a) in California and (b) outside of the state; include 
data on projected and actual sales in both markets and how they compare with competitive products, 
if any exist. 



O 
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SECTION X. Recommendations 

Review all information gathered with this inventory (as well as all other available information 
relevant to the project) and use the space below to make one or more recommendations regarding the 
future of the project: 




so 



Software Development Partnership Program 

SOFTWARE USERS SURVEY 



Far West Laboratory for Educational Research and Development April 1991 

The purpose of this survey is to provide the California Department of Education (CDE) with information 
about the products of four software development partnership projects funded between 1987 and 1990. 
The survey follows CDE guidelines for the California Educational Technology Assessment Project 
(CETAP) being conducted by the Far West Laboratory. 

The form should be completed by each person in a school who has implemented one or 
more of these software products in classroom instruction. A separate form should be 
completed for each product implemented. (Please duplicate additional copies as needed) 

Please compete the survey(s) and return not later than June 14, 1991 to the Far West Laboratory at 
the address below. If any questions or problems arise while completing the questionnaire please contact 
John Cradler, FWL Program Director for Educational Technology and Policy: 

John Cradler Phone: (415)565-3018 

Far West Laboratory FAX: (415) 565-3012 

730 Harrison Street AppleLink: K1686 

San Francisco, CA 94107-1242 CSUNet: jcradle 



A. Background Information 

1. School: District: . 

Address: 

City: Zip: County: 

2. Person completing this form: Title: 

Telephone: Date: / / 

Technology Grant Recipient? Yes No 



3. Software Product Used (circle one number): 

MECC World Geograph 1 

Jostens Explorations in Middle School Science: The Physical Science Program 2 

GTV: A Geographic Perspective on American History 3 

SCIENCE 2000 (field test version) 4 



4. Course/Subject of product application? (fill-in) 



5. 



Estimate the number of students using the product in each grade level: 



# of Students 



K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Adult 































6. Were you aware that the California Department of Education partially supported development 
of this software product? (circle one) 



Yes 1 

No 2 




91 



How did you learn about the software program? (circle all that apply) 

Advertisement/catalog from software producer/vendor 

Sales representative of software producer/vendor 

School principal 

School department head 

School librarian/media/technology specialist 

Teacher colleague at school 

Teacher acquaintance from another school 

Listing in technology newsletter (such as CUE) 

Article in professional journal or newsletter 

District curriculum coordinator/specialist 

District media/technology specialist 

County office curriculum consultant 

County office media/technology consultant 

California Technology Project Regional Consortium representative .... 

Leadership Academy Demo 

California Department of Education 

At professional conference 

TRIE (Technology Resources in Education) database on CSUNet 

Other (fill-in) 

Who bought the software? (fill-in) 



9. Why did you decide to try the software in your class? (circle all that apply) 

The program relates to a curriculum need of my class or classes 

My students get more out of instruction when they use technology programs 

I heard that the software program helped students understand the content/concept 

My colleagues and I want to improve instruction dealing with the topic 

I like using technology programs in my classes 

My students like using technology programs in class 

My principal or department head wanted me to try out the program 

The information about the program made it sound very good 

I am looking for ways to increase student interest in the topic 

It was highly recommended by the person who told me about it 

I was requested to do so by an administrator 

Other (fill-in) 



..1 

..2 

..3 

..4 

..5 

..6 

..7 

..8 

.9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 



.1 

..2 

,3 

.4 

,.5 

.6 

,.7 

,8 

.9 

10 

11 

12 



B. Staff Development 

L In your opinion, is prior computer use necessary to use this product? (circle one) 



Yes 1 

No 2 



2. Have you received any staff development in the use of the software product? (circle one) 



If yes, indicate the source of the staff development (circle all that apply): 



School staff (mentor teacher, media/technology specialist, etc.) 1 

District office staff (media/technology coordinator, etc.) 2 

County Office of Education staff 3 

California Technology Project regional consortium staff 4 

O 2 

ERIC 
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The publisher or distributor of the software product 5 

Other (fill-in) 6 

Was the staff development adequate? (circle one) 

Yes 1 

No 2 



if yes, describe the staff development: 



If no, briefly explain why staff development was not adequate: 



3. What staff development is necessary to use the product effectively? 



Describe: 



4. What training do you think is essential for a teacher to effectively integrate the product into the 
instructional program? 

Describe: 




3 



3 



3 



C. Curriculum Content and Technology Applications 

1. How adequately does the program cover the targeted subject Not at all Moderately Completely 

matter? (circle one) 1 2 3 4 5 

2. What is the availability of the equipment needed to run the product? (circle Y for yes and N for no) 

a. I have all the equipment needed in my classroom Y N 

b. My school has the equipment needed for teachers to share Y N 

c. My school cannot afford the equipment needed Y N 

d. The equipment is easy for most teachers to use Y N 

e. The equipment is cumbersome and difficult for most teachers to use Y N 



D. Instructional Features . 

1. How effective are the special instructional features? 

For each of the features that is incorporated in the product, circle the number that matches your 
rating of its effectiveness in promoting student learning (leave blank if the feature is not part of the 
product): 

Features Not a 

Feature 

a. Adaptability (Can any components be modified by teachers?) I 

b. Flexibility (Can the product be used by individuals, small or large groups?) I 

c. Special Needs (Can it be used by students with special needs?) I 

d. Interactivity (Can students use the product interactively?) 1 

e. Motivational Features (Does the product promote student interest?) 1 

f. Student Problem Solving (Does it provide problem solving opportunities?) 1 

g. Remediation Capability (Does the product help students “catch-up”?) 1 

h. Reinforcement (Does it reinforce other instruction?) 1 

i. Testing (Does it include tests or diagnostic/prescriptive component?) 1 

Other Features (Describe and rate other special features of the product) 

j- 1 

k. : ■ 1 

E. Software Product Evaluation 



1. How is the product used? (circle all that apply) 

Teacher Presentation tool 1 

By students in a large group setting 2 

By students in a small group setting 3 

By individual students 4 

2. How much of the product have you used? (Circle one) 

Less than half of the entire curriculum package 1 

More than half but less than 80 percent of the entire curriculum package 2 

Almost all of the curriculum package 3 

The entire curriculum package 4 

The entire curriculum package more than once 5 

The entire curriculum package more than once with different classes 6 



Moderately AVery 
Effective Effective 
Feature 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 



2 3 4 5 

2 3 4 5 



O 



5 



3. How many times is the product used in your classroom: 

per day per week per month. 

4. How many teachers in your school use the product in your school? 



Note: The remaining items in this section are adapted from the guidelines for evaluating 
various types of educational technology programs prepared for the CDE by the California 
Instructional Video Clearinghouse and the California Computer Software Clearinghouse; 
Guidelines for Computer Software in California Schools and the Guidelines for Computer- 
Interactive Videodisc in California Schools. 



5. How effective is the instructional design of the software product? Circle the number that matches 
your rating of how well the program accomplishes each of the criterion statements: 



Unacceptable 


Acceptable 


Good 


Desirable 


Exemplary 


It does not work 


It works but 


It works 


It does more 


It does all 


with my students 


it's nothing 
special 


quite well 


than expected 


it’s supposed to do 
and then some 


1 


2 


3 


4 


5 



a. Content follows a logical development that students can follow 12 3 

b. Program provides students with positive, appropriate responses 12 3 

c. Students are actively engaged in the program 1 2 3 

d. Program stimulates critical thinking and problem solving 12 3 

e. Presentation of concepts and ideas is better than textbook on subject 12 3 

f. Program is suitable for both individual and group use 1 2 3 

g. Students can use program effectively without extensive teacher direction 12 3 

h. Multiple paths are provided for individual students to meet objectives 12 3 

i. Program permits simple, frame-by-frame access for students who need it 12 3 

j. Beginning users are helped by well-developed, pre-programmed sequences ...1 2 3 

k. Innovative approaches encourage creativity by users 12 3 

l. Program encourages users to locate and use additional learning resources 12 3 

m. Help screens provide clear, understandable explanations 1 2 3 

n. Pace of the program is appropriate for the grade level of students 12 3 

o. Graphic cues support instructional goals and help navigate through program ..1 2 3 



4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 

4 5 n/a 



O 

ERIC 



6 
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6. How effective are the “interactivity” features of the program? Circle the number that matches your 
rating of how well students are able to interact with the program — individually and in groups: 



Unacceptable 


Acceptable 


Good 


Desirable 


Exemplary 


It does not work 


It works but 


It works 


It does more 


It does all 


with my students 


it’s nothing 
special 


quite well 


than expected 


it's supposed to 
and then some 


1 


2 


3 


4 


5 



a. Students can easily and independently operate the program 12 3 4 

b. Indexing is thorough, simple to use, and accurate 12 3 4 

c. Menus or icons allow students direct access to specific parts of program ... 1 2 3 4 

d. Program allows students to alter responses before they are processed 12 3 4 

e. The teacher or the student can determine the pace through the program 12 3 4 

f. Lesson sequences are programmed to branch based on student responses... 1 2 3 4 

g. Program presents harder or easier content according to student responses ..1 2 3 4 

h. Students can go back through the program to review 12 3 4 



5 n/a 
5 n/a 
5 n/a 
5 n/a 
5 n/a 
5 n/a 
5 n/a 
5 n/a 



7. How well does the software product match the curriculum? Circle the number that matches your 
rating of how well the program supports the California Curriculum Frameworks according to the 
following scale: 



Unacceptable 


Acceptable 


Good 


Desirable 


Exemplary 


1 


2 


3 


4 


5 



a. Program content supports specific curriculum objectives in Framework 1 

b. Content supports one or more instructional themes in Framework 1 

c. Program content integrates instructional themes specified in Framework ... 1 

d. There are curriculum-related sequences ready for teacher use 1 



2 3 4 5 n/a 

2 3 4 5 n/a 

2 3 4 5 n/a 

2 3 4 5 n/a 



8. How interesting is the software to students? Circle the number that matches your rating of how 
interesting your students find the program: 



Unacceptable 


Acceptable 


Good 


Desirable 


Exemplary 


It does not work 


It works but 


It works 


It does more 


It does all 


with my students 


it’s nothing 
special 


quite well 


than expected 


it’s supposed to 
and then some 


1 


2 


- 3 


4 


5 



a. The interest level of the program content is well suited to the students 12 3 4 

b. Program content and format stimulate student curiosity and imagination.... 1 2 3 .4 

c. Students are eager to use the program repeatedly and share it with friends .12 3 4 

d. Program provides for various learning modalities (visual, auditory, etc.).... 1 2 3 4 



5 n/a 
5 n/a 
5 n/a 
5 n/a 



9. How do you rate the overall instructional value of the software product? (circle one) 



Very poor; I would not recommend it to other teachers 1 

Poor, I would be quite hesitant to recommend it to other teachers 2 

Fair; I would recommend it with some reservation to other teachers 3 

Good; I would recommend it to other teachers 4 

Superior, I would recommend it to very highly to other teachers 5 



O 



7 



37 



10. Are you aware of similar products? (circle one) 



Yes i 

no ZZZZZZZZZZ‘ZZZZZZZZ.2 



If yes, please name (give publisher, title, and price): 



11. How cost-effective is the software product? That is, was the money spent on this software a 
worthwhile investment? Compare what your students have learned when using the software 
product with the learning of similar groups of students using conventional print materials, such as 
textbooks, (circle one) 



I really don’t know if there is any difference 1 

Conventional print materials are much more cost effective 2 

Conventional print materials are a little more cost effective 3 

Print materials and the software product are about equally cost effective 4 

The software product is a little more cost effective 5 

The software product is much more cost effective 6 

12. What price did your (school, district) pay for this product? $ 

13. Whas the program worth what was paid? (circle one): 

Yes 1 

No *****”””*2 
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